Oct 30 00:31:40.954972 ip-10-0-132-84 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 00:31:40.954984 ip-10-0-132-84 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 00:31:40.954991 ip-10-0-132-84 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 00:31:40.955214 ip-10-0-132-84 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 00:31:50.833650 ip-10-0-132-84 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 00:31:50.833658 ip-10-0-132-84 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 99efdd1783eb4816b4dec8be21e66253 -- Oct 30 00:33:40.769072 ip-10-0-132-84 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:33:41.240219 ip-10-0-132-84 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:33:41.240219 ip-10-0-132-84 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:33:41.240219 ip-10-0-132-84 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:33:41.240219 ip-10-0-132-84 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:33:41.245666 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245563 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:41.245666 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245665 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:41.245666 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245669 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:41.245666 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245674 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245679 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245683 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245686 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245689 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245693 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245695 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245698 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245701 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245719 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245722 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245725 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245728 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245730 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245733 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245736 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245739 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245742 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245744 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245747 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:41.245802 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245749 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245752 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245754 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245757 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245759 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245762 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245764 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245771 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245774 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245777 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245779 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245782 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245785 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245788 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245792 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245795 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245797 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245800 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:41.246272 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245803 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245806 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245809 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245812 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245814 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245817 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245820 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245822 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245825 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245827 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245830 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245832 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245835 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245838 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245840 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245843 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245847 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245849 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245852 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245854 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:41.246697 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245857 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245860 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245863 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245866 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245868 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245871 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245874 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245877 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245880 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245882 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245885 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245887 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245890 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245893 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245895 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245898 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245900 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245903 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245905 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245908 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:41.247182 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245911 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245914 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245917 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245919 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245922 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245924 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245927 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245929 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245933 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245938 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245940 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245943 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245946 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245948 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245951 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245955 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245958 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245960 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245963 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:41.247747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245967 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245969 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245972 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245974 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245977 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.245979 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246817 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246824 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246827 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246830 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246833 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246835 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246838 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246840 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246843 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246846 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246849 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246851 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246854 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246856 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:41.248248 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246859 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246861 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246864 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246867 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246869 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246872 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246874 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246876 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246879 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246881 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246884 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246887 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246890 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246893 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246896 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246898 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246903 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246907 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246910 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246913 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:41.248776 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246916 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246918 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246942 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246945 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246949 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246952 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246956 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246959 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246962 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246965 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246967 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246970 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246973 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246975 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246978 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246980 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246983 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246985 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246988 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:41.249251 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246990 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246993 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246995 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.246998 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247000 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247004 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247006 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247009 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247012 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247014 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247017 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247021 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247025 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247028 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247030 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247033 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247035 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247038 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247040 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:41.249718 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247043 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247045 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247048 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247054 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247056 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247059 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247061 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247064 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247066 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247069 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247072 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247074 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247076 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247079 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247082 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247085 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247087 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247090 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247092 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:41.250239 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247095 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247098 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247100 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247103 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247105 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247108 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247110 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247113 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247115 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247117 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247120 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247122 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247124 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247127 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247130 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247672 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247680 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247683 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247686 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247688 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:41.250738 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247691 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247695 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247699 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247702 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247719 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247721 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247724 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247727 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247729 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247732 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247735 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247737 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247740 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247743 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247746 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247749 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247751 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247754 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247757 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:41.251219 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247760 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247763 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247765 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247768 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247770 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247772 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247775 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247777 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247780 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247783 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247786 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247789 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247791 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247793 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247796 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247798 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247801 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247803 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247806 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247808 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:41.251686 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247810 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247813 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247815 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247817 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247820 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247823 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247826 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247828 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247830 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247833 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247835 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247838 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247840 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247842 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247845 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247847 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247850 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247853 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247855 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247857 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:41.252274 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247860 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247863 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247866 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247868 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247871 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247873 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247876 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247878 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247881 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247883 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247887 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247890 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247893 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247895 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247898 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247902 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247905 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247908 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247910 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:41.252799 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247912 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247915 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247918 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247920 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247923 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247925 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247928 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247930 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247933 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247935 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247938 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247941 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247943 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247946 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247948 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247951 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247954 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247957 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247959 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247962 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:41.253252 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247964 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247967 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.247969 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249080 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249096 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249105 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249110 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249115 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249119 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249123 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249128 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249131 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249134 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249138 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249141 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249144 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249148 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249150 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249154 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249157 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249160 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249164 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249167 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249170 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:33:41.253784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249173 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249177 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249181 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249184 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249187 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249191 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249194 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249197 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249200 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249203 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249207 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249212 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249215 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249218 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249221 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249224 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249227 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249232 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249236 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249239 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249242 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249245 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249249 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249253 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249256 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:33:41.254385 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249259 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249262 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249265 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249268 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249271 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249274 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249279 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249282 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249286 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249290 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249293 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249296 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249299 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249303 2574 flags.go:64] FLAG: --help="false" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249306 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249310 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249313 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249316 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249320 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249323 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249326 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249329 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249332 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:33:41.255043 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249335 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249338 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249341 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249344 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249347 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249350 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249353 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249356 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249359 2574 flags.go:64] FLAG: --lock-file="" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249362 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249365 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249368 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249374 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249377 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249380 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249384 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249387 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249391 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249394 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249397 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249402 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249405 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249409 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249412 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249415 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:33:41.255596 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249418 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249421 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249424 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249427 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249430 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249438 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249441 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249444 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249448 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249451 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249454 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249457 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249460 2574 flags.go:64] FLAG: --port="10250" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249463 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249466 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d5e485531f6b2708" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249469 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249472 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249476 2574 flags.go:64] FLAG: --register-node="true" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249478 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249482 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249485 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249488 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249494 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249498 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249501 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249504 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:33:41.256308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249507 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249510 2574 flags.go:64] FLAG: --runonce="false" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249512 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249516 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249519 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249522 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249525 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249528 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249531 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249535 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249538 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249541 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249546 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249549 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249552 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249555 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249558 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249563 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249566 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249569 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249573 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249577 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249579 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249582 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249586 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:33:41.256993 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249589 2574 flags.go:64] FLAG: --v="2" Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249594 2574 flags.go:64] FLAG: --version="false" Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249598 2574 flags.go:64] FLAG: --vmodule="" Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249604 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.249608 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249765 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249769 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249772 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249774 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249777 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249780 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249782 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249785 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249788 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249791 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249793 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249797 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249799 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249802 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249806 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:41.257599 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249809 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249812 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249815 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249819 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249823 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249826 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249829 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249832 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249834 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249837 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249839 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249842 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249845 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249848 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249850 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249854 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249856 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249859 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249861 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:41.258115 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249864 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249867 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249869 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249872 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249875 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249877 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249879 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249884 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249888 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249891 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249894 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249896 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249900 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249902 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249905 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249907 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249910 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249912 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249915 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:41.258581 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249917 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249920 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249922 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249925 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249927 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249930 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249933 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249935 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249937 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249941 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249944 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249946 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249949 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249951 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249954 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249956 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249959 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249961 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249964 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249967 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:41.259067 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249969 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249973 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249976 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249978 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249980 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249984 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249987 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249989 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249992 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249994 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249997 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.249999 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250002 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250004 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250006 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250009 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250011 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250014 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250016 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250019 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:41.259593 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250022 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250026 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250028 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250030 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250033 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250035 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250038 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250041 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250043 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250045 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250048 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250050 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250053 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.250082 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250126 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:41.260124 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250131 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250134 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250137 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250142 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250144 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250149 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250153 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250156 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250159 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250161 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250164 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250166 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250169 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250171 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250174 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250177 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250179 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250182 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250187 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:41.260487 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250189 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250192 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250194 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250197 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250200 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250202 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250205 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250207 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250210 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250213 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250215 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250217 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250220 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250223 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250226 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250229 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250233 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250235 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250238 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250240 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:41.260978 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250243 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250246 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250248 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250251 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250253 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250256 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250258 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250261 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250264 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250266 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250269 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250273 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250276 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250278 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250281 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250283 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250286 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250288 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250291 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250293 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:41.261469 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250296 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250298 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250301 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250304 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250306 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250309 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250312 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250315 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250319 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250321 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250324 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250326 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250329 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250331 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250334 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250336 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250338 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250341 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250343 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:41.261974 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250346 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250348 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250351 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250354 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250358 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250361 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250364 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250367 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250369 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250372 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250376 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250380 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250383 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250386 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250389 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250391 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250394 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250397 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250400 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:41.262431 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250402 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250406 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250410 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250413 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250415 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250418 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250420 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.250423 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.258211 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.258231 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.258249 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.258254 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258311 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258315 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258319 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258321 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258324 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258327 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258330 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:41.262898 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258332 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258335 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258337 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258340 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258342 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258346 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258349 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258351 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258354 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258356 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258359 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258361 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258364 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258368 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258373 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258376 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258378 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258381 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258383 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:41.263358 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258386 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258388 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258391 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258394 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258397 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258400 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258403 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258405 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258408 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258410 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258413 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258416 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258420 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258423 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258426 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258429 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258432 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258434 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258437 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:41.263839 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258440 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258442 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258444 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258447 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258449 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258452 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258454 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258457 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258459 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258462 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258465 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258468 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258470 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258473 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258475 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258479 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258482 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258485 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258487 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:41.264282 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258490 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258492 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258494 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258497 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258499 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258502 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258505 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258507 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258510 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258512 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258515 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258517 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258520 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258522 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258524 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258527 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258529 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258532 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258534 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:41.264778 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258536 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258539 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258541 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258544 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258547 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258550 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258552 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258555 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258557 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258560 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258563 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258565 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258568 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258570 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258573 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258575 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258578 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258580 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258583 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258586 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:41.265265 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258588 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258591 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258594 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.258625 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258785 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258790 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258793 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258796 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258800 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258804 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258807 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258810 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258812 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258815 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258817 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258820 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:41.265747 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258822 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258825 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258828 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258830 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258833 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258836 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258839 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258841 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258844 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258846 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258849 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258851 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258854 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258856 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258858 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258861 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258863 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258866 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258868 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258871 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:41.266144 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258873 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258876 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258878 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258880 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258883 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258885 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258888 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258890 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258892 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258895 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258897 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258900 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258902 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258904 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258907 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258909 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258912 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258915 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258917 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:41.266651 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258920 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258923 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258926 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258928 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258930 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258933 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258935 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258938 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258940 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258943 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258945 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258948 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258951 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258953 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258956 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258959 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258962 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258964 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258967 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258970 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:41.267120 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258972 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258975 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258977 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258980 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258982 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258985 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258987 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258989 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258993 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258995 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.258998 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259001 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259004 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259007 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259009 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259012 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259014 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259017 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259019 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259022 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:41.267606 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259024 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259027 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259029 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259032 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259034 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259036 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259039 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259041 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259044 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259048 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259051 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259054 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259057 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259059 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259062 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:41.268114 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.259086 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259119 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259122 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259125 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259127 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259130 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259140 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259144 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259147 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259149 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259152 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259154 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259157 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259160 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259162 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259164 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259167 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259169 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259172 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:41.268479 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259174 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259177 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259179 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259181 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259184 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259187 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259189 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259191 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259194 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259196 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259199 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259201 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259204 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259206 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259208 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259211 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259213 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259216 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259218 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:41.269098 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259220 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259226 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259229 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259231 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259234 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259236 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259238 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259241 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259243 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259246 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259248 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259251 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259253 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259256 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259258 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259263 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259266 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259268 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259270 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259273 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:41.269549 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259275 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259277 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259280 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259282 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259285 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259287 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259290 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259292 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259295 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259297 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259299 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259302 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259304 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259306 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259309 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259312 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259315 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259317 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259320 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259322 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:41.270189 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259324 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259327 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259329 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259332 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259334 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259338 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259340 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259344 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259347 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259349 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259352 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259354 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259357 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259360 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259362 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259365 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259367 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259370 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259373 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259375 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:41.271014 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259378 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259380 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259382 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259385 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259390 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259394 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259396 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259400 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:41.259402 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.259458 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.261947 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.263057 2574 server.go:1014] "Starting client certificate rotation" Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.263155 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.263196 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.266110 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:33:41.271502 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.267683 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:33:41.283029 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.283000 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:33:41.287243 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.287222 2574 log.go:25] "Validated CRI v1 image API" Oct 30 00:33:41.288215 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.288191 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:33:41.293660 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.293627 2574 fs.go:135] Filesystem UUIDs: map[73615d2a-4279-4ca9-9f39-4f9c936c230d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e97e6e66-acd8-421c-a221-6ea22b52c37d:/dev/nvme0n1p3] Oct 30 00:33:41.293660 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.293656 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 00:33:41.299220 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.299087 2574 manager.go:217] Machine: {Timestamp:2026-10-30 00:33:41.29759702 +0000 UTC m=+0.385934711 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103972 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22a6cfa14ea2d2c30f5f19a2cae160 SystemUUID:ec22a6cf-a14e-a2d2-c30f-5f19a2cae160 BootID:99efdd17-83eb-4816-b4de-c8be21e66253 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e7:e1:f3:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e7:e1:f3:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:e7:25:8a:7a:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:33:41.299861 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.299849 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:33:41.299960 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.299947 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:33:41.302820 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.302779 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:33:41.303011 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.302821 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:33:41.303129 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.303017 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:33:41.303129 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.303029 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:33:41.303129 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.303077 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:33:41.303129 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.303105 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:33:41.304391 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.304376 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:33:41.309132 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.309110 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:33:41.309298 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.309286 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:33:41.310182 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.310169 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:33:41.310236 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.310191 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:33:41.310236 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.310207 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:33:41.310236 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.310221 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:33:41.310236 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.310235 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:33:41.310665 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.310635 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:33:41.313344 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.313321 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:33:41.321203 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.321182 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:33:41.322160 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.322145 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:33:41.322201 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.322178 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:33:41.322402 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.322387 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:33:41.322455 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.322410 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:33:41.322455 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.322418 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:33:41.322455 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.322426 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:33:41.322455 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.322434 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:33:41.322455 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.322442 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:33:41.323232 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.323210 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:33:41.323232 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.323230 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:33:41.323232 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.323239 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:33:41.323393 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.323246 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:33:41.323393 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.323255 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:33:41.323393 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.323265 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:33:41.324009 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.323991 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:33:41.324009 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.324009 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:33:41.328353 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.328322 2574 server.go:1263] "Started kubelet" Oct 30 00:33:41.328499 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.328430 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:33:41.328559 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.328532 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:33:41.328733 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.328462 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:33:41.329328 ip-10-0-132-84 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:33:41.332800 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.332776 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:33:41.336061 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.336042 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:33:41.339628 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.339592 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 00:33:41.341918 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.341896 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:33:41.343572 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.343554 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:33:41.343654 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.343636 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:41.344726 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344501 2574 factory.go:55] Registering systemd factory Oct 30 00:33:41.344726 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344518 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:33:41.344726 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344629 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:33:41.344726 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344647 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:33:41.344964 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344790 2574 factory.go:153] Registering CRI-O factory Oct 30 00:33:41.344964 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344801 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 00:33:41.344964 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344839 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:33:41.344964 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344849 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:33:41.344964 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344869 2574 factory.go:103] Registering Raw factory Oct 30 00:33:41.344964 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.344883 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 00:33:41.345237 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.345022 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:33:41.345237 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.345032 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:33:41.345912 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.345890 2574 manager.go:319] Starting recovery of all containers Oct 30 00:33:41.355561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.355537 2574 manager.go:324] Recovery completed Oct 30 00:33:41.358760 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.358730 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-84.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 00:33:41.358957 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.358936 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:33:41.359042 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.359020 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-99vh8" Oct 30 00:33:41.359889 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.358722 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-84.ec2.internal.18e3276bf9b814ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-84.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-84.ec2.internal,},FirstTimestamp:2026-10-30 00:33:41.328164046 +0000 UTC m=+0.416501736,LastTimestamp:2026-10-30 00:33:41.328164046 +0000 UTC m=+0.416501736,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-84.ec2.internal,}" Oct 30 00:33:41.360079 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.360067 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:41.361861 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.361845 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:41.361957 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.361877 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:41.361957 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.361891 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:41.362512 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.362498 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:33:41.362512 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.362511 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:41.362650 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.362511 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:33:41.362650 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.362579 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:33:41.363727 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.363697 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:41.363801 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.363739 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:41.363801 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.363750 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:41.364909 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.364897 2574 policy_none.go:50] "Start" Oct 30 00:33:41.364946 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.364914 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:33:41.364946 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.364924 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:33:41.366419 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.366398 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-99vh8" Oct 30 00:33:41.367586 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.367573 2574 policy_none.go:44] "Start" Oct 30 00:33:41.370347 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.370321 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 00:33:41.377432 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.377331 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-84.ec2.internal.18e3276bfbba4487 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-84.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-84.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-84.ec2.internal,},FirstTimestamp:2026-10-30 00:33:41.361861767 +0000 UTC m=+0.450199464,LastTimestamp:2026-10-30 00:33:41.361861767 +0000 UTC m=+0.450199464,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-84.ec2.internal,}" Oct 30 00:33:41.378050 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.378025 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 00:33:41.378050 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.378028 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 00:33:41.401369 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.401346 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.412456 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.415047 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.415080 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.415089 2574 server.go:87] "Starting device plugin registration server" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.415323 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.415333 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.415434 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.415526 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.415535 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.416314 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.416603 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:41.423915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.423064 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:33:41.428280 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.424480 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:33:41.428280 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.424507 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:33:41.428280 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.424528 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:33:41.428280 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.424563 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 00:33:41.441458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.441430 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:41.515494 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.515428 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:41.516488 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.516471 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:41.516550 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.516503 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:41.516550 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.516518 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:41.516629 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.516551 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.525759 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.525718 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal"] Oct 30 00:33:41.525841 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.525801 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:41.526757 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.526741 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:41.526831 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.526773 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:41.526831 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.526786 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:41.529058 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.529046 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:41.529220 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.529206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.529259 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.529237 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:41.529888 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.529865 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:41.530000 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.529899 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:41.530000 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.529916 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:41.530000 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.529926 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:41.530000 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.529953 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:41.530000 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.529968 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:41.532203 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.532184 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.532296 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.532218 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:41.533037 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.533021 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:41.533120 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.533050 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:41.533120 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.533064 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:41.542219 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.542202 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.542280 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.542225 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-84.ec2.internal\": node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:41.546091 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.546072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.546157 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.546098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7678942b8a9336cefbe126b25b5ab55-config\") pod \"kube-apiserver-proxy-ip-10-0-132-84.ec2.internal\" (UID: \"e7678942b8a9336cefbe126b25b5ab55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.546157 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.546126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.561921 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.561898 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-84.ec2.internal\" not found" node="ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.564856 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.564834 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-84.ec2.internal\" not found" node="ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.586366 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.586343 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:41.647146 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.647109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.647316 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.647156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.647316 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.647174 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7678942b8a9336cefbe126b25b5ab55-config\") pod \"kube-apiserver-proxy-ip-10-0-132-84.ec2.internal\" (UID: \"e7678942b8a9336cefbe126b25b5ab55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.647316 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.647226 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7678942b8a9336cefbe126b25b5ab55-config\") pod \"kube-apiserver-proxy-ip-10-0-132-84.ec2.internal\" (UID: \"e7678942b8a9336cefbe126b25b5ab55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.647316 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.647231 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.647316 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.647231 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.686971 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.686936 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:41.788031 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.787949 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:41.867163 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.867131 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.870298 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:41.870269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:33:41.889011 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.888972 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:41.989577 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:41.989521 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:42.090150 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:42.090060 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:42.190694 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:42.190643 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:42.263225 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.263190 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 00:33:42.263625 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.263363 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:33:42.263625 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.263379 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:33:42.291547 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:42.291500 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:42.372231 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.372159 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:28:41 +0000 UTC" deadline="2028-05-08 15:38:42.618497393 +0000 UTC" Oct 30 00:33:42.372231 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.372191 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13359h5m0.246309135s" Oct 30 00:33:42.392080 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:42.392046 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:42.411154 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:42.411121 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32199d49f1d2f12cbdbf36311255bf8c.slice/crio-75b04f1a82854b483f4ade164edb481a99811cb1f85b1002ef1c3d6097410cde WatchSource:0}: Error finding container 75b04f1a82854b483f4ade164edb481a99811cb1f85b1002ef1c3d6097410cde: Status 404 returned error can't find the container with id 75b04f1a82854b483f4ade164edb481a99811cb1f85b1002ef1c3d6097410cde Oct 30 00:33:42.411391 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:42.411362 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7678942b8a9336cefbe126b25b5ab55.slice/crio-3fd39b3321ac2a3cb53f2481875fb203d4f0129e37582d94910d6ad339881821 WatchSource:0}: Error finding container 3fd39b3321ac2a3cb53f2481875fb203d4f0129e37582d94910d6ad339881821: Status 404 returned error can't find the container with id 3fd39b3321ac2a3cb53f2481875fb203d4f0129e37582d94910d6ad339881821 Oct 30 00:33:42.419786 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.419770 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:33:42.428411 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.428340 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" event={"ID":"e7678942b8a9336cefbe126b25b5ab55","Type":"ContainerStarted","Data":"3fd39b3321ac2a3cb53f2481875fb203d4f0129e37582d94910d6ad339881821"} Oct 30 00:33:42.429265 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.429243 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" event={"ID":"32199d49f1d2f12cbdbf36311255bf8c","Type":"ContainerStarted","Data":"75b04f1a82854b483f4ade164edb481a99811cb1f85b1002ef1c3d6097410cde"} Oct 30 00:33:42.492306 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:42.492264 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:42.574005 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.573967 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:42.575274 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.575258 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:42.575335 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.575295 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:42.575335 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.575309 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:42.591238 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.591203 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:42.592995 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:42.592974 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:33:42.675742 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.675703 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:42.745689 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.745640 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:33:42.771806 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.771773 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:33:42.773087 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.773069 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:33:42.798107 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:42.796526 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:33:43.245519 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.245475 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:43.311347 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.311023 2574 apiserver.go:52] "Watching apiserver" Oct 30 00:33:43.342700 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.342667 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:33:43.343133 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.343104 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-f92b8","openshift-network-diagnostics/network-check-target-6ncmx","openshift-ovn-kubernetes/ovnkube-node-qgg9k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal","openshift-multus/network-metrics-daemon-87vfm","openshift-network-operator/iptables-alerter-5whwd","kube-system/konnectivity-agent-jz5dq","openshift-cluster-node-tuning-operator/tuned-md6gg","openshift-dns/node-resolver-62mqm","kube-system/global-pull-secret-syncer-kprr7","openshift-image-registry/node-ca-zbzqk","kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal","openshift-multus/multus-additional-cni-plugins-zxlcf"] Oct 30 00:33:43.343273 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.343145 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 00:33:43.345701 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.345673 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.347955 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.347930 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.351127 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.351090 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.354066 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.353646 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.356149 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356124 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.356604 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.356725 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75bd79f7-bef7-44d5-9257-8048cc570d2c-iptables-alerter-script\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.356725 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75bd79f7-bef7-44d5-9257-8048cc570d2c-host-slash\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.356725 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-socket-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.356725 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-registration-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.356725 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-sys-fs\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.356725 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxgk\" (UniqueName: \"kubernetes.io/projected/4667c989-ad40-4b50-9d15-c341751fff71-kube-api-access-7wxgk\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.356725 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9b5\" (UniqueName: \"kubernetes.io/projected/75bd79f7-bef7-44d5-9257-8048cc570d2c-kube-api-access-cf9b5\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.357109 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.357109 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.356762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-device-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.358536 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.358516 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.360939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.360918 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:43.363619 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.363573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.365208 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.365188 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:33:43.365349 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.365240 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:33:43.365349 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.365299 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:33:43.365597 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.365383 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:33:43.365597 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.365485 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:33:43.365597 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.365522 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:33:43.365727 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.365624 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qpxdw\"" Oct 30 00:33:43.366709 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.366688 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:43.366823 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.366781 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:33:43.369250 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.369217 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.372177 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.371888 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:33:43.372177 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.372005 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:33:43.372177 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.372101 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:33:43.372177 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.372143 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:33:43.372447 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.372304 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:33:43.372447 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.372422 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x59g5\"" Oct 30 00:33:43.372902 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.372881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:43.373045 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.372960 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:33:43.375014 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.374951 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:33:43.375014 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.374963 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:33:43.375014 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.375011 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ltmbf\"" Oct 30 00:33:43.375250 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.375044 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:33:43.375308 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.375283 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:43.375365 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.375350 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:33:43.379660 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.379157 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qff2g\"" Oct 30 00:33:43.379660 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.379267 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:33:43.379660 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.379321 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk5pr\"" Oct 30 00:33:43.379660 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.379330 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:33:43.379660 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.379344 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:33:43.379660 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.379422 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:33:43.379660 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.379434 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s9968\"" Oct 30 00:33:43.380453 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.380419 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2pzq\"" Oct 30 00:33:43.386041 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.386012 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:33:43.386156 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.386061 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:33:43.386156 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.386143 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:33:43.386257 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.386164 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:33:43.389471 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.389448 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:33:43.390127 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.390108 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:33:43.390219 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.390184 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:33:43.390268 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.390255 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b4svw\"" Oct 30 00:33:43.407796 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.407766 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:33:43.407956 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.407920 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gnqsn" Oct 30 00:33:43.407956 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.407940 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gnqsn" Oct 30 00:33:43.446189 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.446163 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:33:43.457577 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-socket-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.457785 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-registration-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.457785 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-sys-fs\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.457785 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.457785 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457746 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-registration-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.457785 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.458079 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457797 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-sys-fs\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.458079 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvgx\" (UniqueName: \"kubernetes.io/projected/c0349f41-063d-4636-963b-4b88924b3670-kube-api-access-fjvgx\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.458079 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-bin\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.458079 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-env-overrides\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.458079 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457941 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cnibin\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.458079 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.457988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-tmp\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.458079 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-serviceca\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.458436 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-netns\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.458436 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-socket-dir-parent\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.458436 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458126 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-socket-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.458436 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d03985-22a9-42a8-a195-638cb40e9e2c-hosts-file\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.458436 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h72b5\" (UniqueName: \"kubernetes.io/projected/16d03985-22a9-42a8-a195-638cb40e9e2c-kube-api-access-h72b5\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.458436 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75bd79f7-bef7-44d5-9257-8048cc570d2c-iptables-alerter-script\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.458436 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lbj\" (UniqueName: \"kubernetes.io/projected/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-kube-api-access-t4lbj\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.458436 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-kubernetes\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.458436 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njq4r\" (UniqueName: \"kubernetes.io/projected/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-kube-api-access-njq4r\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458455 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-systemd-units\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-modprobe-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458594 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-system-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-conf-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-lib-modules\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-k8s-cni-cncf-io\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-netns\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-host\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-etc-tuned\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.458939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-konnectivity-ca\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-etc-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.458991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-log-socket\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459023 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75bd79f7-bef7-44d5-9257-8048cc570d2c-iptables-alerter-script\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459018 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-script-lib\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbh9\" (UniqueName: \"kubernetes.io/projected/d1d6ac87-947e-44dd-9a24-1b1df8755f06-kube-api-access-4cbh9\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-system-cni-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459216 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75bd79f7-bef7-44d5-9257-8048cc570d2c-host-slash\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459273 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75bd79f7-bef7-44d5-9257-8048cc570d2c-host-slash\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459335 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-multus\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-device-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-run\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-slash\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.459541 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-config\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459559 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-device-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-var-lib-kubelet\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxzz\" (UniqueName: \"kubernetes.io/projected/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-kube-api-access-gbxzz\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-agent-certs\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-systemd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-ovn\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.459956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-bin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-hostroot\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.460258 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460099 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9b5\" (UniqueName: \"kubernetes.io/projected/75bd79f7-bef7-44d5-9257-8048cc570d2c-kube-api-access-cf9b5\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460307 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-conf\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460338 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-kubelet\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-multus-certs\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxgk\" (UniqueName: \"kubernetes.io/projected/4667c989-ad40-4b50-9d15-c341751fff71-kube-api-access-7wxgk\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460456 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-etc-kubernetes\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d03985-22a9-42a8-a195-638cb40e9e2c-tmp-dir\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpqf\" (UniqueName: \"kubernetes.io/projected/61e5f3a5-6460-4b3e-a423-70d0000dd706-kube-api-access-xvpqf\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460706 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-var-lib-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-node-log\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.460977 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.460986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cni-binary-copy\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-daemon-config\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-host\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461256 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461290 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-kubelet-config\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cnibin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461356 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-netd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysconfig\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461477 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-systemd\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-sys\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-kubelet\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-dbus\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:43.461695 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.461684 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.473961 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.473892 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:33:43.473961 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.473920 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:33:43.473961 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.473972 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tcw96\"" Oct 30 00:33:43.487696 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.487652 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:33:43.492597 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.492563 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxgk\" (UniqueName: \"kubernetes.io/projected/4667c989-ad40-4b50-9d15-c341751fff71-kube-api-access-7wxgk\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.492781 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.492645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9b5\" (UniqueName: \"kubernetes.io/projected/75bd79f7-bef7-44d5-9257-8048cc570d2c-kube-api-access-cf9b5\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.562895 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.562790 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-run\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.562895 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.562836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-slash\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.562895 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.562864 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-config\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.562895 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.562890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.562923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-var-lib-kubelet\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.562950 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxzz\" (UniqueName: \"kubernetes.io/projected/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-kube-api-access-gbxzz\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.562948 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-slash\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.562975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-agent-certs\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563025 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-run\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-systemd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563061 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-var-lib-kubelet\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563085 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-systemd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-ovn\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563121 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-ovn\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-bin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.563162 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563206 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-hostroot\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563207 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-bin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.563239 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:44.063209157 +0000 UTC m=+3.151546865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:43.563248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-hostroot\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-conf\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-kubelet\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-multus-certs\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563348 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-kubelet\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-etc-kubernetes\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563387 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-multus-certs\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d03985-22a9-42a8-a195-638cb40e9e2c-tmp-dir\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563415 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-etc-kubernetes\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpqf\" (UniqueName: \"kubernetes.io/projected/61e5f3a5-6460-4b3e-a423-70d0000dd706-kube-api-access-xvpqf\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563457 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-var-lib-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-node-log\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563485 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-conf\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.564150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563577 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563581 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-config\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cni-binary-copy\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563611 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-node-log\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-daemon-config\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563638 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-var-lib-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563586 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-host\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563729 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-host\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563745 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563761 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d03985-22a9-42a8-a195-638cb40e9e2c-tmp-dir\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-kubelet-config\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cnibin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-netd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563920 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565071 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563937 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-kubelet-config\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysconfig\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563977 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-netd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.563992 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-systemd\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564032 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cnibin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-sys\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564039 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysconfig\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564063 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-systemd\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-kubelet\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-dbus\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564109 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-sys\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564139 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-kubelet\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvgx\" (UniqueName: \"kubernetes.io/projected/c0349f41-063d-4636-963b-4b88924b3670-kube-api-access-fjvgx\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-bin\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564220 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-env-overrides\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.565941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564227 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cni-binary-copy\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cnibin\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564260 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-dbus\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564265 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-tmp\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-serviceca\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564309 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-bin\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-netns\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-socket-dir-parent\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cnibin\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d03985-22a9-42a8-a195-638cb40e9e2c-hosts-file\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h72b5\" (UniqueName: \"kubernetes.io/projected/16d03985-22a9-42a8-a195-638cb40e9e2c-kube-api-access-h72b5\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564414 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-socket-dir-parent\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564423 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lbj\" (UniqueName: \"kubernetes.io/projected/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-kube-api-access-t4lbj\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-kubernetes\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564481 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njq4r\" (UniqueName: \"kubernetes.io/projected/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-kube-api-access-njq4r\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-systemd-units\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.566646 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-modprobe-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564543 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d03985-22a9-42a8-a195-638cb40e9e2c-hosts-file\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-system-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564574 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-conf-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-lib-modules\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564630 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-k8s-cni-cncf-io\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-netns\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-host\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564674 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-netns\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-etc-tuned\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564721 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-konnectivity-ca\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564773 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-etc-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-log-socket\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-script-lib\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564830 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-kubernetes\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.567458 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564852 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbh9\" (UniqueName: \"kubernetes.io/projected/d1d6ac87-947e-44dd-9a24-1b1df8755f06-kube-api-access-4cbh9\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564879 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-system-cni-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564885 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-systemd-units\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564898 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-lib-modules\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-multus\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.564914 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564955 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-system-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.564963 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret podName:4d0d2d6e-c723-4393-92e5-65c3e8b4f713 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:44.064950463 +0000 UTC m=+3.153288142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret") pod "global-pull-secret-syncer-kprr7" (UID: "4d0d2d6e-c723-4393-92e5-65c3e8b4f713") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564975 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-modprobe-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.564998 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-conf-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565032 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-system-cni-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565042 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-multus\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565081 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-k8s-cni-cncf-io\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565141 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-netns\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565153 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565203 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-host\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565291 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-log-socket\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565518 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-env-overrides\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.568561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565654 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-serviceca\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565692 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-etc-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565754 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-script-lib\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.565869 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-konnectivity-ca\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.566137 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.566241 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.566287 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-daemon-config\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.567141 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.567611 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-agent-certs\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.568034 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-etc-tuned\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.569373 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.568582 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-tmp\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.588750 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.588697 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:43.588750 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.588752 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:43.588957 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.588768 2574 projected.go:196] Error preparing data for projected volume kube-api-access-4nfmc for pod openshift-network-diagnostics/network-check-target-6ncmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:43.588957 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:43.588852 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc podName:e868adcd-9f67-4757-b2e1-9cb31a968d02 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:44.088828907 +0000 UTC m=+3.177166611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4nfmc" (UniqueName: "kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc") pod "network-check-target-6ncmx" (UID: "e868adcd-9f67-4757-b2e1-9cb31a968d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:43.594518 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.594487 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxzz\" (UniqueName: \"kubernetes.io/projected/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-kube-api-access-gbxzz\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.594766 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.594727 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lbj\" (UniqueName: \"kubernetes.io/projected/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-kube-api-access-t4lbj\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.594967 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.594945 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72b5\" (UniqueName: \"kubernetes.io/projected/16d03985-22a9-42a8-a195-638cb40e9e2c-kube-api-access-h72b5\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.595621 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.595595 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvgx\" (UniqueName: \"kubernetes.io/projected/c0349f41-063d-4636-963b-4b88924b3670-kube-api-access-fjvgx\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.595621 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.595602 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbh9\" (UniqueName: \"kubernetes.io/projected/d1d6ac87-947e-44dd-9a24-1b1df8755f06-kube-api-access-4cbh9\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.595899 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.595864 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpqf\" (UniqueName: \"kubernetes.io/projected/61e5f3a5-6460-4b3e-a423-70d0000dd706-kube-api-access-xvpqf\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:43.597219 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.597195 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njq4r\" (UniqueName: \"kubernetes.io/projected/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-kube-api-access-njq4r\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.662929 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.662664 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:33:43.675128 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:43.675090 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bd79f7_bef7_44d5_9257_8048cc570d2c.slice/crio-b17adb0349ceea2e111330a8a2f949c63fabfdbe0498e869c87baa71c97e1b12 WatchSource:0}: Error finding container b17adb0349ceea2e111330a8a2f949c63fabfdbe0498e869c87baa71c97e1b12: Status 404 returned error can't find the container with id b17adb0349ceea2e111330a8a2f949c63fabfdbe0498e869c87baa71c97e1b12 Oct 30 00:33:43.676395 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.676372 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:33:43.682771 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.682745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f92b8" Oct 30 00:33:43.692203 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:43.690912 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4667c989_ad40_4b50_9d15_c341751fff71.slice/crio-0f2055fed2cb973618817524e7254b82e8f0c51e45dc2647e342bcffcfed7b82 WatchSource:0}: Error finding container 0f2055fed2cb973618817524e7254b82e8f0c51e45dc2647e342bcffcfed7b82: Status 404 returned error can't find the container with id 0f2055fed2cb973618817524e7254b82e8f0c51e45dc2647e342bcffcfed7b82 Oct 30 00:33:43.692603 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.692581 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:33:43.701222 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:43.701189 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d6ac87_947e_44dd_9a24_1b1df8755f06.slice/crio-f5b85cc09e62a99dc5348a46cd9498ad8bb1ad820458fb599f43f71cb4434f1c WatchSource:0}: Error finding container f5b85cc09e62a99dc5348a46cd9498ad8bb1ad820458fb599f43f71cb4434f1c: Status 404 returned error can't find the container with id f5b85cc09e62a99dc5348a46cd9498ad8bb1ad820458fb599f43f71cb4434f1c Oct 30 00:33:43.703563 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.703543 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-62mqm" Oct 30 00:33:43.705003 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:43.704958 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0349f41_063d_4636_963b_4b88924b3670.slice/crio-1d4633a4aea9b0b1018ed3b9c0500f953936d4b5a8837c49841bfd7e224e3063 WatchSource:0}: Error finding container 1d4633a4aea9b0b1018ed3b9c0500f953936d4b5a8837c49841bfd7e224e3063: Status 404 returned error can't find the container with id 1d4633a4aea9b0b1018ed3b9c0500f953936d4b5a8837c49841bfd7e224e3063 Oct 30 00:33:43.715136 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.714950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:33:43.717179 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:43.717147 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d03985_22a9_42a8_a195_638cb40e9e2c.slice/crio-0ef8519f2f57921e78d0cbc43a84adcf78d125514cecde68e63f5373a96592cb WatchSource:0}: Error finding container 0ef8519f2f57921e78d0cbc43a84adcf78d125514cecde68e63f5373a96592cb: Status 404 returned error can't find the container with id 0ef8519f2f57921e78d0cbc43a84adcf78d125514cecde68e63f5373a96592cb Oct 30 00:33:43.721568 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.721544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:43.726995 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:43.726964 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa5c5ce_6d41_4294_ae89_c5f54b35dfe9.slice/crio-585e6d02764a61f1550beda8a8c4ce2b3d98a82f42e071b49cf16d8a001fff64 WatchSource:0}: Error finding container 585e6d02764a61f1550beda8a8c4ce2b3d98a82f42e071b49cf16d8a001fff64: Status 404 returned error can't find the container with id 585e6d02764a61f1550beda8a8c4ce2b3d98a82f42e071b49cf16d8a001fff64 Oct 30 00:33:43.732884 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:43.732853 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7f74c1_965a_4ccc_a8f0_2dbd1770908e.slice/crio-43760e3abdb60e7366ff855e69ec0a5fc1fab5633b4cca56d55ff465a1aebfb0 WatchSource:0}: Error finding container 43760e3abdb60e7366ff855e69ec0a5fc1fab5633b4cca56d55ff465a1aebfb0: Status 404 returned error can't find the container with id 43760e3abdb60e7366ff855e69ec0a5fc1fab5633b4cca56d55ff465a1aebfb0 Oct 30 00:33:43.750827 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.750793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:33:43.762166 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.762139 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:33:43.763372 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:43.763336 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b47aade_5b0f_4ab4_ac11_e71acc17a9cb.slice/crio-8cf5dfebd102a6423e17241ba07ad036ecb8f1feaa362239b6a5ccaec78763e4 WatchSource:0}: Error finding container 8cf5dfebd102a6423e17241ba07ad036ecb8f1feaa362239b6a5ccaec78763e4: Status 404 returned error can't find the container with id 8cf5dfebd102a6423e17241ba07ad036ecb8f1feaa362239b6a5ccaec78763e4 Oct 30 00:33:43.774468 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:33:43.774438 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dd358a_8c7b_4a22_8181_fa6b5aa931a9.slice/crio-206cfd82e95202a2549277a2ac890c7a953febcad2aefe0736d466cfaf839611 WatchSource:0}: Error finding container 206cfd82e95202a2549277a2ac890c7a953febcad2aefe0736d466cfaf839611: Status 404 returned error can't find the container with id 206cfd82e95202a2549277a2ac890c7a953febcad2aefe0736d466cfaf839611 Oct 30 00:33:43.777910 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:43.777889 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:44.070293 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.070254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:44.070425 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.070352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:44.070558 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:44.070539 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:44.070629 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:44.070615 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret podName:4d0d2d6e-c723-4393-92e5-65c3e8b4f713 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:45.070595821 +0000 UTC m=+4.158933521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret") pod "global-pull-secret-syncer-kprr7" (UID: "4d0d2d6e-c723-4393-92e5-65c3e8b4f713") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:44.072389 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:44.071097 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:44.072389 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:44.071160 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:45.071140841 +0000 UTC m=+4.159478526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:44.171313 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.170916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:44.171313 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:44.171304 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:44.171618 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:44.171332 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:44.171618 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:44.171347 2574 projected.go:196] Error preparing data for projected volume kube-api-access-4nfmc for pod openshift-network-diagnostics/network-check-target-6ncmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:44.171618 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:44.171421 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc podName:e868adcd-9f67-4757-b2e1-9cb31a968d02 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:45.171400589 +0000 UTC m=+4.259738275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4nfmc" (UniqueName: "kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc") pod "network-check-target-6ncmx" (UID: "e868adcd-9f67-4757-b2e1-9cb31a968d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:44.411964 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.411808 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:28:43 +0000 UTC" deadline="2028-06-06 23:17:39.389435315 +0000 UTC" Oct 30 00:33:44.411964 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.411880 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14062h43m54.977559969s" Oct 30 00:33:44.439131 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.439087 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f92b8" event={"ID":"d1d6ac87-947e-44dd-9a24-1b1df8755f06","Type":"ContainerStarted","Data":"f5b85cc09e62a99dc5348a46cd9498ad8bb1ad820458fb599f43f71cb4434f1c"} Oct 30 00:33:44.447434 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.447386 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" event={"ID":"e7678942b8a9336cefbe126b25b5ab55","Type":"ContainerStarted","Data":"7ad366bc8c1ab8c0a53e6d2497d4f235b8d02f53a6b2127fb46afce64e0c99f9"} Oct 30 00:33:44.451155 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.451120 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" event={"ID":"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9","Type":"ContainerStarted","Data":"206cfd82e95202a2549277a2ac890c7a953febcad2aefe0736d466cfaf839611"} Oct 30 00:33:44.452918 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.452852 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zbzqk" event={"ID":"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9","Type":"ContainerStarted","Data":"585e6d02764a61f1550beda8a8c4ce2b3d98a82f42e071b49cf16d8a001fff64"} Oct 30 00:33:44.454662 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.454591 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-62mqm" event={"ID":"16d03985-22a9-42a8-a195-638cb40e9e2c","Type":"ContainerStarted","Data":"0ef8519f2f57921e78d0cbc43a84adcf78d125514cecde68e63f5373a96592cb"} Oct 30 00:33:44.460520 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.460416 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" event={"ID":"4667c989-ad40-4b50-9d15-c341751fff71","Type":"ContainerStarted","Data":"0f2055fed2cb973618817524e7254b82e8f0c51e45dc2647e342bcffcfed7b82"} Oct 30 00:33:44.462979 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.462891 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5whwd" event={"ID":"75bd79f7-bef7-44d5-9257-8048cc570d2c","Type":"ContainerStarted","Data":"b17adb0349ceea2e111330a8a2f949c63fabfdbe0498e869c87baa71c97e1b12"} Oct 30 00:33:44.468513 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.467825 2574 generic.go:356] "Generic (PLEG): container finished" podID="32199d49f1d2f12cbdbf36311255bf8c" containerID="39a119fdadc0a9c2b43c47719fc2d88a4123cc38b018e685cc7ce2fcc32c1e43" exitCode=0 Oct 30 00:33:44.468513 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.467913 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" event={"ID":"32199d49f1d2f12cbdbf36311255bf8c","Type":"ContainerDied","Data":"39a119fdadc0a9c2b43c47719fc2d88a4123cc38b018e685cc7ce2fcc32c1e43"} Oct 30 00:33:44.481025 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.480906 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" event={"ID":"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb","Type":"ContainerStarted","Data":"8cf5dfebd102a6423e17241ba07ad036ecb8f1feaa362239b6a5ccaec78763e4"} Oct 30 00:33:44.494259 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.494156 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jz5dq" event={"ID":"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e","Type":"ContainerStarted","Data":"43760e3abdb60e7366ff855e69ec0a5fc1fab5633b4cca56d55ff465a1aebfb0"} Oct 30 00:33:44.501322 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.501235 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-md6gg" event={"ID":"c0349f41-063d-4636-963b-4b88924b3670","Type":"ContainerStarted","Data":"1d4633a4aea9b0b1018ed3b9c0500f953936d4b5a8837c49841bfd7e224e3063"} Oct 30 00:33:44.536527 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:44.535497 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" podStartSLOduration=1.535477312 podStartE2EDuration="1.535477312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:44.490922771 +0000 UTC m=+3.579260477" watchObservedRunningTime="2026-10-30 00:33:44.535477312 +0000 UTC m=+3.623815016" Oct 30 00:33:45.079504 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.079264 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:45.079504 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.079346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:45.079814 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.079508 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:45.079814 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.079572 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:47.079554 +0000 UTC m=+6.167891695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:45.080108 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.080010 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:45.080108 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.080070 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret podName:4d0d2d6e-c723-4393-92e5-65c3e8b4f713 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:47.080052914 +0000 UTC m=+6.168390612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret") pod "global-pull-secret-syncer-kprr7" (UID: "4d0d2d6e-c723-4393-92e5-65c3e8b4f713") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:45.180686 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.180643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:45.180950 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.180899 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:45.180950 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.180928 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:45.180950 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.180943 2574 projected.go:196] Error preparing data for projected volume kube-api-access-4nfmc for pod openshift-network-diagnostics/network-check-target-6ncmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:45.181119 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.181006 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc podName:e868adcd-9f67-4757-b2e1-9cb31a968d02 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:47.18098801 +0000 UTC m=+6.269325711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4nfmc" (UniqueName: "kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc") pod "network-check-target-6ncmx" (UID: "e868adcd-9f67-4757-b2e1-9cb31a968d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:45.412409 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.412288 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:28:43 +0000 UTC" deadline="2028-04-29 16:21:31.661271321 +0000 UTC" Oct 30 00:33:45.412409 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.412359 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13143h47m46.248916549s" Oct 30 00:33:45.426912 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.426868 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:45.427128 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.427013 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:33:45.427462 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.427439 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:45.427572 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.427550 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:33:45.427647 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.427633 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:45.427740 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:45.427719 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:33:45.525908 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.525865 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" event={"ID":"32199d49f1d2f12cbdbf36311255bf8c","Type":"ContainerStarted","Data":"ca7f19dcc24517a5d8dcfc0c6b5be10e9937d6c846e44cc10860e65c73c6ce26"} Oct 30 00:33:45.569679 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:45.569611 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" podStartSLOduration=2.569590001 podStartE2EDuration="2.569590001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:45.569451642 +0000 UTC m=+4.657789346" watchObservedRunningTime="2026-10-30 00:33:45.569590001 +0000 UTC m=+4.657927700" Oct 30 00:33:46.546854 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:46.546817 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" event={"ID":"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9","Type":"ContainerStarted","Data":"aedcc55647313538671986be4d904a0b5f449e35bb0b2e2055c60be9dac18590"} Oct 30 00:33:47.096105 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:47.095403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:47.096105 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:47.095484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:47.096105 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.095657 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:47.096105 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.095724 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:51.095704351 +0000 UTC m=+10.184042046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:47.096105 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.095910 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:47.096105 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.095992 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret podName:4d0d2d6e-c723-4393-92e5-65c3e8b4f713 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:51.09596975 +0000 UTC m=+10.184307447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret") pod "global-pull-secret-syncer-kprr7" (UID: "4d0d2d6e-c723-4393-92e5-65c3e8b4f713") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:47.196424 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:47.196380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:47.196653 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.196633 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:47.196709 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.196665 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:47.196709 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.196680 2574 projected.go:196] Error preparing data for projected volume kube-api-access-4nfmc for pod openshift-network-diagnostics/network-check-target-6ncmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:47.196839 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.196745 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc podName:e868adcd-9f67-4757-b2e1-9cb31a968d02 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:51.196727061 +0000 UTC m=+10.285064756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4nfmc" (UniqueName: "kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc") pod "network-check-target-6ncmx" (UID: "e868adcd-9f67-4757-b2e1-9cb31a968d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:47.424972 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:47.424894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:47.425143 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.425029 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:33:47.425474 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:47.425449 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:47.425600 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.425578 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:33:47.425670 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:47.425643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:47.425733 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:47.425716 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:33:47.557593 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:47.557548 2574 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="aedcc55647313538671986be4d904a0b5f449e35bb0b2e2055c60be9dac18590" exitCode=0 Oct 30 00:33:47.558140 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:47.557629 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" event={"ID":"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9","Type":"ContainerDied","Data":"aedcc55647313538671986be4d904a0b5f449e35bb0b2e2055c60be9dac18590"} Oct 30 00:33:48.567525 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:48.567443 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jz5dq" event={"ID":"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e","Type":"ContainerStarted","Data":"3074edb1dd3b843e56baf0cdb7bf7f8bc4e7b69c2131fc5dc5ecb1d02e98fdcd"} Oct 30 00:33:48.632226 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:48.632145 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jz5dq" podStartSLOduration=2.5815019230000003 podStartE2EDuration="6.63212604s" totalImagesPullingTime="4.050624117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:48.632055858 +0000 UTC m=+7.720393558" watchObservedRunningTime="2026-10-30 00:33:48.63212604 +0000 UTC m=+7.720463743" Oct 30 00:33:49.427623 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:49.427590 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:49.427813 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:49.427738 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:33:49.427891 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:49.427842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:49.427971 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:49.427944 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:33:49.428078 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:49.428064 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:49.428183 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:49.428149 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:33:49.578555 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:49.578081 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zbzqk" event={"ID":"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9","Type":"ContainerStarted","Data":"44d55ea437d4a1be3019cddc1268fd7bd7b18977e9ea77ee21925623ff9b1729"} Oct 30 00:33:49.617615 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:49.617558 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zbzqk" podStartSLOduration=3.513365711 podStartE2EDuration="8.617543725s" totalImagesPullingTime="5.104178014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:49.616978411 +0000 UTC m=+8.705316112" watchObservedRunningTime="2026-10-30 00:33:49.617543725 +0000 UTC m=+8.705881425" Oct 30 00:33:50.585690 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:50.585634 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" event={"ID":"4667c989-ad40-4b50-9d15-c341751fff71","Type":"ContainerStarted","Data":"e1c48b8d7a6bba72f3285e80bad6e276c44d1a32122bc2b0b6a9790469bd95c9"} Oct 30 00:33:51.136911 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.136843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:51.137023 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.136913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:51.137023 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.137007 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:51.137143 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.137024 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:51.137143 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.137075 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret podName:4d0d2d6e-c723-4393-92e5-65c3e8b4f713 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:59.137056458 +0000 UTC m=+18.225394138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret") pod "global-pull-secret-syncer-kprr7" (UID: "4d0d2d6e-c723-4393-92e5-65c3e8b4f713") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:51.137143 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.137097 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:59.137086504 +0000 UTC m=+18.225424181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:51.238051 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.238021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:51.238227 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.238205 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:51.238268 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.238239 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:51.238268 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.238254 2574 projected.go:196] Error preparing data for projected volume kube-api-access-4nfmc for pod openshift-network-diagnostics/network-check-target-6ncmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:51.238343 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.238311 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc podName:e868adcd-9f67-4757-b2e1-9cb31a968d02 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:59.238292772 +0000 UTC m=+18.326630454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4nfmc" (UniqueName: "kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc") pod "network-check-target-6ncmx" (UID: "e868adcd-9f67-4757-b2e1-9cb31a968d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:51.267995 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.267969 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:33:51.421785 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.421668 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:33:51.267991575Z","UUID":"6729654a-eb79-450f-a020-15b1dcf019c1","Handler":null,"Name":"","Endpoint":""} Oct 30 00:33:51.423300 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.423283 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:33:51.423390 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.423306 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:33:51.426429 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.426035 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:51.426429 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.426134 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:33:51.426968 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.426777 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:51.426968 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.426926 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:33:51.426968 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.426950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:51.427136 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:51.427056 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:33:51.593429 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.593384 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f92b8" event={"ID":"d1d6ac87-947e-44dd-9a24-1b1df8755f06","Type":"ContainerStarted","Data":"dee16f7696849f24238ef11c7f7b82bd9940b0f17530f7ca3938f352cce60419"} Oct 30 00:33:51.599896 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.599858 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" event={"ID":"4667c989-ad40-4b50-9d15-c341751fff71","Type":"ContainerStarted","Data":"9f7a68d73b4b8b539978001848230e87a9c19e91c99b242e3e753d8079f5266f"} Oct 30 00:33:51.642574 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.642485 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-f92b8" podStartSLOduration=2.960952638 podStartE2EDuration="10.642473913s" totalImagesPullingTime="7.681521275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:51.642136197 +0000 UTC m=+10.730473897" watchObservedRunningTime="2026-10-30 00:33:51.642473913 +0000 UTC m=+10.730811613" Oct 30 00:33:51.678601 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.678555 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:51.679745 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:51.679511 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:33:52.605187 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:52.605103 2574 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="f6194cc2f72607ebd8e2f6cd1fb9f9bcb03744784595a73fd41583c746022552" exitCode=0 Oct 30 00:33:52.605639 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:52.605183 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" event={"ID":"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9","Type":"ContainerDied","Data":"f6194cc2f72607ebd8e2f6cd1fb9f9bcb03744784595a73fd41583c746022552"} Oct 30 00:33:52.608643 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:52.608614 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-62mqm" event={"ID":"16d03985-22a9-42a8-a195-638cb40e9e2c","Type":"ContainerStarted","Data":"a24ad2d49d6b510dd7c16a04bd9c20b2f7c8352048519a8d815dd62331a4b274"} Oct 30 00:33:52.613144 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:52.613079 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" event={"ID":"4667c989-ad40-4b50-9d15-c341751fff71","Type":"ContainerStarted","Data":"a665a1cf529b0a6a590de36e6984533bfa1bbea4e56a3c9efc0ddb0248fcd59b"} Oct 30 00:33:52.701125 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:52.700898 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" podStartSLOduration=3.387331375 podStartE2EDuration="11.700883518s" totalImagesPullingTime="8.313552143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:52.700285212 +0000 UTC m=+11.788622939" watchObservedRunningTime="2026-10-30 00:33:52.700883518 +0000 UTC m=+11.789221218" Oct 30 00:33:52.744156 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:52.744096 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-62mqm" podStartSLOduration=3.433467321 podStartE2EDuration="11.744080319s" totalImagesPullingTime="8.310612998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:52.743987928 +0000 UTC m=+11.832325629" watchObservedRunningTime="2026-10-30 00:33:52.744080319 +0000 UTC m=+11.832418020" Oct 30 00:33:53.425699 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:53.425673 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:53.425855 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:53.425827 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:53.426081 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:53.426029 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:53.426167 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:53.426078 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:33:53.426167 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:53.425949 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:33:53.426247 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:53.426182 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:33:53.618379 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:53.618306 2574 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="7163be8b300e2d16f5e8ad84f7a52bb7568127e293c58d05794b1768b03e82eb" exitCode=0 Oct 30 00:33:53.618834 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:53.618391 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" event={"ID":"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9","Type":"ContainerDied","Data":"7163be8b300e2d16f5e8ad84f7a52bb7568127e293c58d05794b1768b03e82eb"} Oct 30 00:33:53.620449 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:53.620394 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5whwd" event={"ID":"75bd79f7-bef7-44d5-9257-8048cc570d2c","Type":"ContainerStarted","Data":"5081d4b3fa940eccf83c0fcb0704b30ae55f133f926897878bd08fab02028443"} Oct 30 00:33:53.704142 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:53.704095 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5whwd" podStartSLOduration=4.667237993 podStartE2EDuration="12.70408056s" totalImagesPullingTime="8.036842567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:53.703758778 +0000 UTC m=+12.792096478" watchObservedRunningTime="2026-10-30 00:33:53.70408056 +0000 UTC m=+12.792418263" Oct 30 00:33:54.624507 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:54.624421 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-md6gg" event={"ID":"c0349f41-063d-4636-963b-4b88924b3670","Type":"ContainerStarted","Data":"a0893e525db3188f6dacf033b589bc914f77531ef8cd03bd52c2108fb5371c18"} Oct 30 00:33:54.628038 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:54.628014 2574 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="326d7e663e9d48db865a527adc8d6db4d5a9508bbc1dfa517bd8ff020dbeae36" exitCode=0 Oct 30 00:33:54.628172 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:54.628097 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" event={"ID":"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9","Type":"ContainerDied","Data":"326d7e663e9d48db865a527adc8d6db4d5a9508bbc1dfa517bd8ff020dbeae36"} Oct 30 00:33:54.667656 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:54.667609 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-md6gg" podStartSLOduration=3.61441527 podStartE2EDuration="13.667596425s" totalImagesPullingTime="10.053181155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:54.666919022 +0000 UTC m=+13.755256727" watchObservedRunningTime="2026-10-30 00:33:54.667596425 +0000 UTC m=+13.755934124" Oct 30 00:33:55.428836 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:55.428811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:55.429030 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:55.428905 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:55.429030 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:55.428919 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:33:55.429030 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:55.429003 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:33:55.429030 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:55.429025 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:55.429239 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:55.429121 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:33:57.425635 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:57.425600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:57.426222 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:57.425749 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:33:57.426222 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:57.426081 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:57.426222 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:57.426153 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:33:57.426222 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:57.426202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:57.426379 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:57.426260 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:33:57.638263 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:57.638017 2574 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="cd8726e8de5035d54f26d73e6afb7f356a21050ef61114d310cec00807b1f4bf" exitCode=0 Oct 30 00:33:57.638421 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:57.638084 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" event={"ID":"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9","Type":"ContainerDied","Data":"cd8726e8de5035d54f26d73e6afb7f356a21050ef61114d310cec00807b1f4bf"} Oct 30 00:33:58.643220 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:58.643187 2574 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="1d1f7dac4e7e8c54c6d860a50864e860b750baaf727d50b9ed24cd8e16da630c" exitCode=0 Oct 30 00:33:58.643514 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:58.643225 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" event={"ID":"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9","Type":"ContainerDied","Data":"1d1f7dac4e7e8c54c6d860a50864e860b750baaf727d50b9ed24cd8e16da630c"} Oct 30 00:33:58.645752 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:58.645732 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" event={"ID":"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb","Type":"ContainerStarted","Data":"b318d0f1c8962cf7cd1d1c3ee9877509df04dfaa0f15339aa54d1e1f14e77ab1"} Oct 30 00:33:58.645939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:58.645797 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" event={"ID":"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb","Type":"ContainerStarted","Data":"86ef1814c6da2e1f9b291e8c6a536ccb30d1db5f033963ad553c95208437d33a"} Oct 30 00:33:58.645939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:58.645829 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" event={"ID":"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb","Type":"ContainerStarted","Data":"6275f119f3ea5e6b4e8f616c2bf4ee18d45764b93e532f289a53c7e458cb91de"} Oct 30 00:33:59.199966 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.199746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:59.200139 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.200009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:59.200139 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.199898 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:59.200139 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.200101 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:59.200139 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.200119 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:15.200095776 +0000 UTC m=+34.288433467 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:59.200372 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.200145 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret podName:4d0d2d6e-c723-4393-92e5-65c3e8b4f713 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:15.200133526 +0000 UTC m=+34.288471217 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret") pod "global-pull-secret-syncer-kprr7" (UID: "4d0d2d6e-c723-4393-92e5-65c3e8b4f713") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:59.301161 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.301130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:59.301315 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.301251 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:59.301315 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.301267 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:59.301315 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.301275 2574 projected.go:196] Error preparing data for projected volume kube-api-access-4nfmc for pod openshift-network-diagnostics/network-check-target-6ncmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:59.301400 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.301317 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc podName:e868adcd-9f67-4757-b2e1-9cb31a968d02 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:15.301306036 +0000 UTC m=+34.389643714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4nfmc" (UniqueName: "kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc") pod "network-check-target-6ncmx" (UID: "e868adcd-9f67-4757-b2e1-9cb31a968d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:59.427815 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.427793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:33:59.427972 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.427883 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:33:59.427972 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.427793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:33:59.427972 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.427920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:33:59.428109 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.427984 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:33:59.428109 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:33:59.428075 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:33:59.650788 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.650749 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" event={"ID":"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb","Type":"ContainerStarted","Data":"6202384ad95fc14e190b430af2e32ef9e119d53f7acb1478c9e6be8c4fb32c2d"} Oct 30 00:33:59.651209 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.650817 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" event={"ID":"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb","Type":"ContainerStarted","Data":"9f660bd30088e253c2a278b6a7ef8416479cdbcff05b6790fb7c139ccc3c860d"} Oct 30 00:33:59.651209 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.650855 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" event={"ID":"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb","Type":"ContainerStarted","Data":"9cc8b926916169dde5a82b4ebb4b40be593c780a2127ddd017a2317ce9f4ebbf"} Oct 30 00:33:59.653902 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.653879 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" event={"ID":"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9","Type":"ContainerStarted","Data":"0f47092e14ea528800710ac0092157b1eb7f2d191478ff34678a8b7acdae5b39"} Oct 30 00:33:59.699947 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:33:59.699904 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zxlcf" podStartSLOduration=7.541052934 podStartE2EDuration="17.699890322s" totalImagesPullingTime="10.158837388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:59.699673776 +0000 UTC m=+18.788011487" watchObservedRunningTime="2026-10-30 00:33:59.699890322 +0000 UTC m=+18.788228022" Oct 30 00:34:00.966118 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:00.966082 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:34:00.966523 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:00.966246 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:34:00.966750 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:00.966733 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:34:01.427577 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:01.427544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:01.427750 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:01.427582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:01.427750 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:01.427556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:01.427750 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:01.427647 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:34:01.427915 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:01.427739 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:34:01.427915 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:01.427804 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:34:01.661567 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:01.661530 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" event={"ID":"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb","Type":"ContainerStarted","Data":"77cf53efae7965e2de19ccddbaee86b030495b0d5ca0c25c7c79cea3bf2d47d3"} Oct 30 00:34:03.428444 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.428413 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:03.429334 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.428549 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:03.429334 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:03.428565 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:34:03.429334 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:03.428632 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:34:03.429334 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.428657 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:03.429334 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:03.428755 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:34:03.669544 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.669498 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" event={"ID":"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb","Type":"ContainerStarted","Data":"ff7781624384470cd9342c79fe4f702d40a8d48e7682ad0d9efef9309a717667"} Oct 30 00:34:03.669775 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.669688 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:03.669775 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.669761 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:03.669892 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.669811 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:03.685838 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.685770 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:03.685952 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.685876 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:03.759915 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:03.759862 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" podStartSLOduration=7.048958775 podStartE2EDuration="21.759850151s" totalImagesPullingTime="14.710891376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:03.759400799 +0000 UTC m=+22.847738511" watchObservedRunningTime="2026-10-30 00:34:03.759850151 +0000 UTC m=+22.848187850" Oct 30 00:34:05.092021 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:05.091846 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kprr7"] Oct 30 00:34:05.092391 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:05.092101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:05.092391 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:05.092185 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:34:05.095326 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:05.095295 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ncmx"] Oct 30 00:34:05.095439 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:05.095398 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:05.095505 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:05.095488 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:34:05.104482 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:05.104456 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87vfm"] Oct 30 00:34:05.104608 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:05.104559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:05.104686 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:05.104665 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:34:06.424784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:06.424747 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:06.425231 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:06.424748 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:06.425231 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:06.424856 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:34:06.425231 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:06.424951 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:34:07.425691 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:07.425659 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:07.426060 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:07.425817 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:34:08.425603 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:08.425575 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:08.425603 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:08.425591 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:08.426185 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:08.425679 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:34:08.426185 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:08.425825 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:34:09.425328 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:09.425246 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:09.425469 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:09.425372 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87vfm" podUID="61e5f3a5-6460-4b3e-a423-70d0000dd706" Oct 30 00:34:10.424979 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:10.424920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:10.424979 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:10.424955 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:10.425419 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:10.425053 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kprr7" podUID="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" Oct 30 00:34:10.425419 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:10.425163 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ncmx" podUID="e868adcd-9f67-4757-b2e1-9cb31a968d02" Oct 30 00:34:10.837932 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:10.837897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:11.025570 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.025536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:11.157763 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.157737 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeReady" Oct 30 00:34:11.157925 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.157874 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 00:34:11.215553 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.215523 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tg442"] Oct 30 00:34:11.219184 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.219163 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.220287 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.220263 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qgjbc"] Oct 30 00:34:11.224694 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.224678 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:11.237173 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.237156 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:34:11.237263 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.237252 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:34:11.237313 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.237274 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-75g57\"" Oct 30 00:34:11.241927 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.241905 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tg442"] Oct 30 00:34:11.250551 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.250530 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgjbc"] Oct 30 00:34:11.337593 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.337558 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:34:11.337593 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.337582 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:34:11.337593 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.337576 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:34:11.337858 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.337655 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pkqjh\"" Oct 30 00:34:11.343110 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.343083 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:11.389575 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.389537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkdb\" (UniqueName: \"kubernetes.io/projected/b43fb340-5191-4f45-b6d9-9916afc9a2fd-kube-api-access-6gkdb\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.389784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.389582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43fb340-5191-4f45-b6d9-9916afc9a2fd-tmp-dir\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.389784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.389610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkcpc\" (UniqueName: \"kubernetes.io/projected/3b703ac5-d445-4378-8d48-eea0ec2563fc-kube-api-access-tkcpc\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:11.389784 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.389719 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fb340-5191-4f45-b6d9-9916afc9a2fd-config-volume\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.389920 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.389786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.389920 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.389868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b703ac5-d445-4378-8d48-eea0ec2563fc-cert\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:11.425638 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.425555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:11.443398 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.443372 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:34:11.443526 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.443512 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q972j\"" Oct 30 00:34:11.456941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.456912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:11.490864 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.490833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.490864 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.490866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b703ac5-d445-4378-8d48-eea0ec2563fc-cert\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:11.491097 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:11.490968 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:11.491097 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.491009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkdb\" (UniqueName: \"kubernetes.io/projected/b43fb340-5191-4f45-b6d9-9916afc9a2fd-kube-api-access-6gkdb\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.491097 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:11.491026 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:11.991012322 +0000 UTC m=+31.079349999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:11.491097 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.491055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43fb340-5191-4f45-b6d9-9916afc9a2fd-tmp-dir\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.491097 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.491073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkcpc\" (UniqueName: \"kubernetes.io/projected/3b703ac5-d445-4378-8d48-eea0ec2563fc-kube-api-access-tkcpc\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:11.491310 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.491107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fb340-5191-4f45-b6d9-9916afc9a2fd-config-volume\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.491403 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.491387 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43fb340-5191-4f45-b6d9-9916afc9a2fd-tmp-dir\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.491644 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.491625 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fb340-5191-4f45-b6d9-9916afc9a2fd-config-volume\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.494723 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.494685 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b703ac5-d445-4378-8d48-eea0ec2563fc-cert\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:11.514530 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.514505 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkcpc\" (UniqueName: \"kubernetes.io/projected/3b703ac5-d445-4378-8d48-eea0ec2563fc-kube-api-access-tkcpc\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:11.514634 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.514582 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkdb\" (UniqueName: \"kubernetes.io/projected/b43fb340-5191-4f45-b6d9-9916afc9a2fd-kube-api-access-6gkdb\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.538627 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.538598 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:11.846320 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.846246 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgjbc"] Oct 30 00:34:11.850934 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:34:11.850904 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b703ac5_d445_4378_8d48_eea0ec2563fc.slice/crio-81ce993d82a5521103059e452631659e6719d192c30a120e60cfd25b30347ffe WatchSource:0}: Error finding container 81ce993d82a5521103059e452631659e6719d192c30a120e60cfd25b30347ffe: Status 404 returned error can't find the container with id 81ce993d82a5521103059e452631659e6719d192c30a120e60cfd25b30347ffe Oct 30 00:34:11.995519 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:11.995485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:11.995683 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:11.995628 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:11.995742 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:11.995690 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.995675431 +0000 UTC m=+32.084013110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:12.425406 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:12.425371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:12.425585 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:12.425407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:12.440924 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:12.440899 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:34:12.441312 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:12.440990 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:34:12.494524 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:12.494498 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78zxw\"" Oct 30 00:34:12.494524 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:12.494497 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-smldl\"" Oct 30 00:34:12.507234 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:12.507203 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:34:12.691057 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:12.690964 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgjbc" event={"ID":"3b703ac5-d445-4378-8d48-eea0ec2563fc","Type":"ContainerStarted","Data":"81ce993d82a5521103059e452631659e6719d192c30a120e60cfd25b30347ffe"} Oct 30 00:34:13.002352 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:13.002277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:13.002499 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:13.002460 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:13.002547 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:13.002540 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:15.002520601 +0000 UTC m=+34.090858306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:14.697594 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:14.697513 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgjbc" event={"ID":"3b703ac5-d445-4378-8d48-eea0ec2563fc","Type":"ContainerStarted","Data":"d9a34535fd74ecc51780cf42419f4a9c28bcb766fc3224e97bc98beb171bef87"} Oct 30 00:34:14.737849 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:14.737792 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qgjbc" podStartSLOduration=1.213961468 podStartE2EDuration="3.737775396s" totalImagesPullingTime="2.523813928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:14.73745765 +0000 UTC m=+33.825795349" watchObservedRunningTime="2026-10-30 00:34:14.737775396 +0000 UTC m=+33.826113097" Oct 30 00:34:14.848894 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:14.848868 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:14.965200 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:14.965097 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:15.017053 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.017029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:15.017199 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:15.017184 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:15.017251 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:15.017242 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:19.017226126 +0000 UTC m=+38.105563804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:15.218867 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.218694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:15.218867 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.218860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:15.219065 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:15.218977 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:15.219065 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:15.219046 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:47.219027513 +0000 UTC m=+66.307365198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : secret "metrics-daemon-secret" not found Oct 30 00:34:15.221065 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.221045 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:15.320074 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.320044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:15.322561 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.322540 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:15.450203 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.450177 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:15.471364 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.471318 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:15.797767 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.797737 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ncmx"] Oct 30 00:34:15.806679 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:34:15.806654 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode868adcd_9f67_4757_b2e1_9cb31a968d02.slice/crio-de6481df9297cfdd627c26a66311bff025dcd73d2fb6219d3670eb8196b3a146 WatchSource:0}: Error finding container de6481df9297cfdd627c26a66311bff025dcd73d2fb6219d3670eb8196b3a146: Status 404 returned error can't find the container with id de6481df9297cfdd627c26a66311bff025dcd73d2fb6219d3670eb8196b3a146 Oct 30 00:34:15.815890 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:15.815867 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kprr7"] Oct 30 00:34:15.819675 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:34:15.819653 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0d2d6e_c723_4393_92e5_65c3e8b4f713.slice/crio-def8ab59d7a2782db2cda69fd22d8fa59a7f2371af36398380fef28a6b345bda WatchSource:0}: Error finding container def8ab59d7a2782db2cda69fd22d8fa59a7f2371af36398380fef28a6b345bda: Status 404 returned error can't find the container with id def8ab59d7a2782db2cda69fd22d8fa59a7f2371af36398380fef28a6b345bda Oct 30 00:34:16.704131 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:16.704090 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ncmx" event={"ID":"e868adcd-9f67-4757-b2e1-9cb31a968d02","Type":"ContainerStarted","Data":"de6481df9297cfdd627c26a66311bff025dcd73d2fb6219d3670eb8196b3a146"} Oct 30 00:34:16.709413 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:16.709372 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kprr7" event={"ID":"4d0d2d6e-c723-4393-92e5-65c3e8b4f713","Type":"ContainerStarted","Data":"def8ab59d7a2782db2cda69fd22d8fa59a7f2371af36398380fef28a6b345bda"} Oct 30 00:34:17.338010 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.337967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:17.380463 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.380434 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7"] Oct 30 00:34:17.402419 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.401585 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" Oct 30 00:34:17.403475 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.403415 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7"] Oct 30 00:34:17.417178 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.417155 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 00:34:17.417500 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.417482 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:17.429643 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.429611 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-th8hp\"" Oct 30 00:34:17.453227 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.453148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:17.513304 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.513263 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lj5vr"] Oct 30 00:34:17.530201 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.530177 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.533151 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.533119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw7zp\" (UniqueName: \"kubernetes.io/projected/87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e-kube-api-access-pw7zp\") pod \"migrator-66df5d496b-pg7p7\" (UID: \"87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" Oct 30 00:34:17.540816 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.540794 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lj5vr"] Oct 30 00:34:17.574932 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.574896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:17.589737 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.589637 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xtmdf-8fqw8"] Oct 30 00:34:17.611958 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.611922 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.613139 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.613115 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xtmdf-8fqw8"] Oct 30 00:34:17.617817 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.617792 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:34:17.617922 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.617859 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:34:17.618016 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.617998 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:34:17.618016 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.617999 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:34:17.618193 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.618097 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c889x\"" Oct 30 00:34:17.634021 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.633985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.634190 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.634037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7zp\" (UniqueName: \"kubernetes.io/projected/87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e-kube-api-access-pw7zp\") pod \"migrator-66df5d496b-pg7p7\" (UID: \"87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" Oct 30 00:34:17.634248 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.634193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.634302 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.634259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68188523-c3a4-4a55-90a8-0ba680aad7e2-crio-socket\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.634302 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.634290 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2dn\" (UniqueName: \"kubernetes.io/projected/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-api-access-jb2dn\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.634397 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.634317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68188523-c3a4-4a55-90a8-0ba680aad7e2-data-volume\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.660032 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.659997 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7zp\" (UniqueName: \"kubernetes.io/projected/87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e-kube-api-access-pw7zp\") pod \"migrator-66df5d496b-pg7p7\" (UID: \"87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" Oct 30 00:34:17.682336 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.682245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:17.718359 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.718329 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 00:34:17.718610 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.718591 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pc46q\"" Oct 30 00:34:17.727829 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.727800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" Oct 30 00:34:17.734855 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.734824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68188523-c3a4-4a55-90a8-0ba680aad7e2-data-volume\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.735040 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.734865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzh7g\" (UniqueName: \"kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.735040 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.734903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.735040 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.734962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.735040 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.734993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.735040 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.735043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.735302 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.735082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.735302 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:17.735089 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:17.735302 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.735114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68188523-c3a4-4a55-90a8-0ba680aad7e2-crio-socket\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.735302 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.735129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2dn\" (UniqueName: \"kubernetes.io/projected/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-api-access-jb2dn\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.735302 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:17.735206 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.235171363 +0000 UTC m=+37.323509056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:17.735559 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:17.735463 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:34:17.735559 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:17.735523 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.235506492 +0000 UTC m=+37.323844187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : configmap "kube-rbac-proxy" not found Oct 30 00:34:17.735677 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.735657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68188523-c3a4-4a55-90a8-0ba680aad7e2-crio-socket\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.735807 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.735774 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68188523-c3a4-4a55-90a8-0ba680aad7e2-data-volume\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.744547 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.744514 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 00:34:17.759745 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.759692 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2dn\" (UniqueName: \"kubernetes.io/projected/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-api-access-jb2dn\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:17.800524 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.800479 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:17.837076 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.835455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.837076 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.835523 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.837076 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.835567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.837076 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.835666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzh7g\" (UniqueName: \"kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.837356 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.837181 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.837590 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.837566 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.837774 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.837758 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.864856 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.864783 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzh7g\" (UniqueName: \"kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:17.911315 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.911278 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:17.938893 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:17.938858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:18.029619 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.029538 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:18.060014 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.059956 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7"] Oct 30 00:34:18.064805 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:34:18.064776 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c5d9af_1ffa_4bcc_a983_ddea4a8cba3e.slice/crio-f3e922f85fb4cd2b947e5c3cbf067473d4f7a20cd2d29d7421c6d37c22cdc7e1 WatchSource:0}: Error finding container f3e922f85fb4cd2b947e5c3cbf067473d4f7a20cd2d29d7421c6d37c22cdc7e1: Status 404 returned error can't find the container with id f3e922f85fb4cd2b947e5c3cbf067473d4f7a20cd2d29d7421c6d37c22cdc7e1 Oct 30 00:34:18.239297 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.239204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:18.239297 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.239257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:18.239527 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:18.239363 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:34:18.239527 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:18.239387 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:18.239527 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:18.239438 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:19.239417793 +0000 UTC m=+38.327755497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : configmap "kube-rbac-proxy" not found Oct 30 00:34:18.239527 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:18.239460 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:19.239449506 +0000 UTC m=+38.327787189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:18.260390 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.260299 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xtmdf-8fqw8"] Oct 30 00:34:18.267822 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:34:18.267792 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e30b08_acdb_4c39_9689_018566263138.slice/crio-cc5b6a01276734b88c6053bdf78ed59fe8f43f248d94c240fd0d79225847bcf2 WatchSource:0}: Error finding container cc5b6a01276734b88c6053bdf78ed59fe8f43f248d94c240fd0d79225847bcf2: Status 404 returned error can't find the container with id cc5b6a01276734b88c6053bdf78ed59fe8f43f248d94c240fd0d79225847bcf2 Oct 30 00:34:18.275468 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.275440 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fqjft"] Oct 30 00:34:18.307690 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.307648 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fqjft"] Oct 30 00:34:18.311758 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.310836 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.326466 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.326432 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 00:34:18.326612 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.326542 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 00:34:18.326612 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.326544 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8qb6v\"" Oct 30 00:34:18.326612 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.326544 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 00:34:18.326787 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.326735 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 00:34:18.440834 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.440791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmr64\" (UniqueName: \"kubernetes.io/projected/041a2fd8-4580-4074-a167-7cb444d727e0-kube-api-access-nmr64\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.441318 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.440886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/041a2fd8-4580-4074-a167-7cb444d727e0-signing-key\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.441318 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.440924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/041a2fd8-4580-4074-a167-7cb444d727e0-signing-cabundle\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.542038 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.541968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/041a2fd8-4580-4074-a167-7cb444d727e0-signing-cabundle\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.542038 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.542036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr64\" (UniqueName: \"kubernetes.io/projected/041a2fd8-4580-4074-a167-7cb444d727e0-kube-api-access-nmr64\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.542311 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.542287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/041a2fd8-4580-4074-a167-7cb444d727e0-signing-key\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.542882 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.542852 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/041a2fd8-4580-4074-a167-7cb444d727e0-signing-cabundle\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.545238 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.545217 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/041a2fd8-4580-4074-a167-7cb444d727e0-signing-key\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.565939 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.565876 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmr64\" (UniqueName: \"kubernetes.io/projected/041a2fd8-4580-4074-a167-7cb444d727e0-kube-api-access-nmr64\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.641624 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.641582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:18.728395 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.719347 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" event={"ID":"72e30b08-acdb-4c39-9689-018566263138","Type":"ContainerStarted","Data":"cc5b6a01276734b88c6053bdf78ed59fe8f43f248d94c240fd0d79225847bcf2"} Oct 30 00:34:18.728395 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.721426 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" event={"ID":"87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e","Type":"ContainerStarted","Data":"f3e922f85fb4cd2b947e5c3cbf067473d4f7a20cd2d29d7421c6d37c22cdc7e1"} Oct 30 00:34:18.731121 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.731002 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:18.852466 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.852382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:18.989819 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:18.989780 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fqjft"] Oct 30 00:34:19.003150 ip-10-0-132-84 kubenswrapper[2574]: W1030 00:34:19.003114 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041a2fd8_4580_4074_a167_7cb444d727e0.slice/crio-b274b478ad62a0b147aa9e6c1411e7639312424d0a4fad4b3bc5b90880edb337 WatchSource:0}: Error finding container b274b478ad62a0b147aa9e6c1411e7639312424d0a4fad4b3bc5b90880edb337: Status 404 returned error can't find the container with id b274b478ad62a0b147aa9e6c1411e7639312424d0a4fad4b3bc5b90880edb337 Oct 30 00:34:19.048387 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:19.048346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:19.048570 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:19.048502 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:19.048570 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:19.048567 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:27.048549272 +0000 UTC m=+46.136886956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:19.186013 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:19.185778 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:19.250251 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:19.250212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:19.250420 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:19.250318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:19.250420 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:19.250374 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:34:19.250420 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:19.250420 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:19.250538 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:19.250463 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:21.250441052 +0000 UTC m=+40.338778765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : configmap "kube-rbac-proxy" not found Oct 30 00:34:19.250538 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:19.250485 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:21.25047484 +0000 UTC m=+40.338812518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:19.726150 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:19.726112 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" event={"ID":"041a2fd8-4580-4074-a167-7cb444d727e0","Type":"ContainerStarted","Data":"b274b478ad62a0b147aa9e6c1411e7639312424d0a4fad4b3bc5b90880edb337"} Oct 30 00:34:20.732005 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:20.731971 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ncmx" event={"ID":"e868adcd-9f67-4757-b2e1-9cb31a968d02","Type":"ContainerStarted","Data":"be960505720d8efb64bfa396d780e9b3788d991a6b555ddc47ac71de95b36d7e"} Oct 30 00:34:20.732504 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:20.732080 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:20.733688 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:20.733652 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" event={"ID":"87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e","Type":"ContainerStarted","Data":"29e3a291c81394973c74a87112fa2d9625dc79151405f7d867c1b865b3876847"} Oct 30 00:34:20.774659 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:20.774594 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6ncmx" podStartSLOduration=34.794698926 podStartE2EDuration="38.774574184s" totalImagesPullingTime="3.979875258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:20.773232676 +0000 UTC m=+39.861570376" watchObservedRunningTime="2026-10-30 00:34:20.774574184 +0000 UTC m=+39.862911889" Oct 30 00:34:21.269064 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:21.269024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:21.269251 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:21.269118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:21.269251 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:21.269171 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:34:21.269251 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:21.269247 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:21.269387 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:21.269257 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:25.269237208 +0000 UTC m=+44.357574900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : configmap "kube-rbac-proxy" not found Oct 30 00:34:21.269387 ip-10-0-132-84 kubenswrapper[2574]: E1030 00:34:21.269293 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:25.269282116 +0000 UTC m=+44.357619793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:21.504861 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:21.504826 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:21.738941 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:21.738903 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" event={"ID":"72e30b08-acdb-4c39-9689-018566263138","Type":"ContainerStarted","Data":"82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0"} Oct 30 00:34:21.780212 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:21.780156 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" podStartSLOduration=2.738098736 podStartE2EDuration="4.78013674s" totalImagesPullingTime="2.042038004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:20.831675907 +0000 UTC m=+39.920013610" watchObservedRunningTime="2026-10-30 00:34:21.78013674 +0000 UTC m=+40.868474440" Oct 30 00:34:21.780365 ip-10-0-132-84 kubenswrapper[2574]: I1030 00:34:21.780262 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" podStartSLOduration=1.75288808 podStartE2EDuration="4.780256952s" totalImagesPullingTime="3.027368872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:21.779139547 +0000 UTC m=+40.867477247" watchObservedRunningTime="2026-10-30 00:34:21.780256952 +0000 UTC m=+40.868594654" Oct 30 00:34:22.019045 ip-10-0-132-84 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 00:34:22.033268 ip-10-0-132-84 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 00:34:22.033528 ip-10-0-132-84 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 00:34:22.033585 ip-10-0-132-84 systemd[1]: kubelet.service: Consumed 2.498s CPU time. Oct 30 00:34:22.045356 ip-10-0-132-84 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:34:22.136079 ip-10-0-132-84 kubenswrapper[4517]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:34:22.136079 ip-10-0-132-84 kubenswrapper[4517]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:34:22.136079 ip-10-0-132-84 kubenswrapper[4517]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:34:22.136079 ip-10-0-132-84 kubenswrapper[4517]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:34:22.138533 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138429 4517 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:22.138533 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138529 4517 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:22.138533 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138533 4517 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:22.138533 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138537 4517 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:22.138533 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138542 4517 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138548 4517 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138553 4517 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138558 4517 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138563 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138570 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138574 4517 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138579 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138582 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138584 4517 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138587 4517 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138595 4517 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138598 4517 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138601 4517 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138603 4517 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138606 4517 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138610 4517 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138615 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138619 4517 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138624 4517 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:22.138815 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138627 4517 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138632 4517 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138636 4517 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138643 4517 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138649 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138654 4517 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138659 4517 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138663 4517 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138667 4517 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138671 4517 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138676 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138680 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138684 4517 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138688 4517 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138692 4517 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138695 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138698 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138701 4517 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138717 4517 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:22.139450 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138720 4517 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138723 4517 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138725 4517 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138728 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138732 4517 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138735 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138739 4517 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138744 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138747 4517 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138751 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138758 4517 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138762 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138767 4517 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138770 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138773 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138776 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138779 4517 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138781 4517 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138784 4517 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:22.140102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138787 4517 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138790 4517 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138793 4517 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138796 4517 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138798 4517 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138801 4517 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138804 4517 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138806 4517 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138810 4517 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138813 4517 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138816 4517 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138819 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138822 4517 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138825 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138827 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138831 4517 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138836 4517 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138841 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138846 4517 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138850 4517 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:22.140690 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138853 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138856 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138858 4517 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138861 4517 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138864 4517 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138867 4517 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138869 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138872 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138874 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138877 4517 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138879 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138881 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138884 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138887 4517 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138890 4517 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138892 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138895 4517 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138897 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138900 4517 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138903 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:22.141429 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138908 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138912 4517 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138919 4517 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.138924 4517 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139924 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139937 4517 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139943 4517 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139948 4517 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139953 4517 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139956 4517 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139959 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139963 4517 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139966 4517 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139969 4517 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139971 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139974 4517 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139976 4517 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139979 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139982 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139985 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:22.142030 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139987 4517 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139990 4517 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139992 4517 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139996 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.139998 4517 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140001 4517 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140003 4517 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140006 4517 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140008 4517 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140011 4517 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140014 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140016 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140020 4517 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140025 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140030 4517 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140034 4517 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140038 4517 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140041 4517 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140045 4517 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140049 4517 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140053 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:22.143011 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140058 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140062 4517 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140066 4517 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140069 4517 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140073 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140075 4517 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140079 4517 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140087 4517 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140093 4517 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140098 4517 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140103 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140107 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140112 4517 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140117 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140122 4517 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140126 4517 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140133 4517 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140137 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140141 4517 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:22.143679 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140144 4517 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140147 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140150 4517 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140156 4517 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140159 4517 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140161 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140164 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140166 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140170 4517 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140172 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140175 4517 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140177 4517 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140180 4517 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140183 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140185 4517 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140188 4517 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140190 4517 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140194 4517 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140196 4517 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:22.144290 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140199 4517 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140202 4517 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140204 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140206 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140209 4517 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140211 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140214 4517 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140216 4517 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140218 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140221 4517 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140224 4517 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140227 4517 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140230 4517 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140233 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140235 4517 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140238 4517 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140242 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140245 4517 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140247 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:22.144981 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140250 4517 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140252 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140255 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140257 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140260 4517 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140262 4517 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140265 4517 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140267 4517 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140270 4517 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140273 4517 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140276 4517 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140279 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140904 4517 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140912 4517 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140916 4517 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140918 4517 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140921 4517 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140924 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140926 4517 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140929 4517 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:22.145563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140932 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140935 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140937 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140939 4517 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140942 4517 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140945 4517 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140948 4517 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140950 4517 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140955 4517 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140958 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140961 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140963 4517 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140967 4517 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140971 4517 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140975 4517 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140980 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140984 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140988 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140992 4517 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140996 4517 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:22.146349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.140999 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141002 4517 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141004 4517 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141007 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141011 4517 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141014 4517 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141016 4517 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141019 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141021 4517 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141025 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141030 4517 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141034 4517 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141038 4517 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141042 4517 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141046 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141051 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141055 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141060 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141066 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:22.147304 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141071 4517 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141082 4517 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141085 4517 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141088 4517 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141090 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141093 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141095 4517 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141099 4517 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141102 4517 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141105 4517 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141108 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141111 4517 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141115 4517 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141119 4517 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141124 4517 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141128 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141133 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141136 4517 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141141 4517 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:22.148207 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141143 4517 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141146 4517 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141148 4517 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141151 4517 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141153 4517 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141156 4517 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141158 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141161 4517 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141163 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141167 4517 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141169 4517 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141172 4517 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141176 4517 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141180 4517 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141187 4517 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141192 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141197 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141201 4517 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141206 4517 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141211 4517 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:22.149039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141216 4517 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141220 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141224 4517 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141227 4517 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141229 4517 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141232 4517 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141235 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141237 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141240 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141242 4517 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141244 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141247 4517 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141259 4517 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141262 4517 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141264 4517 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141267 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141269 4517 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141272 4517 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141275 4517 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.141278 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:22.149825 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141401 4517 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141416 4517 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141423 4517 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141428 4517 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141432 4517 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141436 4517 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141442 4517 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141452 4517 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141455 4517 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141458 4517 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141462 4517 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141465 4517 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141468 4517 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141471 4517 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141474 4517 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141477 4517 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141480 4517 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141483 4517 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141487 4517 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141490 4517 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141493 4517 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141496 4517 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141500 4517 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141504 4517 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:34:22.150401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141507 4517 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141512 4517 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141515 4517 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141519 4517 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141522 4517 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141525 4517 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141528 4517 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141531 4517 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141538 4517 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141543 4517 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141548 4517 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141551 4517 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141554 4517 flags.go:64] FLAG: --enable-server="true" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141558 4517 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141568 4517 flags.go:64] FLAG: --event-burst="100" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141574 4517 flags.go:64] FLAG: --event-qps="50" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141579 4517 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141584 4517 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141589 4517 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141596 4517 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141601 4517 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141605 4517 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141608 4517 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141611 4517 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141614 4517 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:34:22.151388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141617 4517 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141619 4517 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141622 4517 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141625 4517 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141629 4517 flags.go:64] FLAG: --feature-gates="" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141633 4517 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141636 4517 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141639 4517 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141643 4517 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141647 4517 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141650 4517 flags.go:64] FLAG: --help="false" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141654 4517 flags.go:64] FLAG: --hostname-override="ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141659 4517 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141663 4517 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141669 4517 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141675 4517 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141681 4517 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141686 4517 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141691 4517 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141696 4517 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141702 4517 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141725 4517 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141732 4517 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141737 4517 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:34:22.152181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141742 4517 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141747 4517 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141752 4517 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141756 4517 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141760 4517 flags.go:64] FLAG: --lock-file="" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141764 4517 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141770 4517 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141775 4517 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141781 4517 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141784 4517 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141787 4517 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141790 4517 flags.go:64] FLAG: --logging-format="text" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141793 4517 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141796 4517 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141799 4517 flags.go:64] FLAG: --manifest-url="" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141802 4517 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141807 4517 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141810 4517 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141815 4517 flags.go:64] FLAG: --max-pods="110" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141818 4517 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141821 4517 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141824 4517 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141827 4517 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141832 4517 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141837 4517 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:34:22.152980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141841 4517 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141851 4517 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141855 4517 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141857 4517 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141862 4517 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141867 4517 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141872 4517 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141877 4517 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141881 4517 flags.go:64] FLAG: --port="10250" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141886 4517 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141890 4517 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d5e485531f6b2708" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141895 4517 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141900 4517 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141905 4517 flags.go:64] FLAG: --register-node="true" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141909 4517 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141915 4517 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141920 4517 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141925 4517 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141929 4517 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141935 4517 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141941 4517 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141945 4517 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141948 4517 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141951 4517 flags.go:64] FLAG: --runonce="false" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141953 4517 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141957 4517 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141961 4517 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:34:22.153841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141964 4517 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141967 4517 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141970 4517 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141973 4517 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141976 4517 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141979 4517 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141982 4517 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141984 4517 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141987 4517 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141992 4517 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141997 4517 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.141999 4517 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142007 4517 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142013 4517 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142017 4517 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142024 4517 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142027 4517 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142030 4517 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142033 4517 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142036 4517 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142040 4517 flags.go:64] FLAG: --v="2" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142047 4517 flags.go:64] FLAG: --version="false" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142054 4517 flags.go:64] FLAG: --vmodule="" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142060 4517 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142065 4517 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:34:22.154764 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142243 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142250 4517 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142255 4517 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142260 4517 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142264 4517 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142268 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142272 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142277 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142283 4517 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142289 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142294 4517 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142299 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142303 4517 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142307 4517 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142312 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142315 4517 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142332 4517 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142337 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142340 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:22.155490 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142342 4517 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142345 4517 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142348 4517 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142351 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142353 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142356 4517 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142359 4517 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142364 4517 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142368 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142373 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142377 4517 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142380 4517 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142383 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142387 4517 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142391 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142395 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142400 4517 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142405 4517 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142409 4517 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142414 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:22.156208 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142419 4517 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142424 4517 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142429 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142433 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142438 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142442 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142447 4517 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142451 4517 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142455 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142461 4517 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142466 4517 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142471 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142475 4517 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142479 4517 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142483 4517 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142488 4517 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142492 4517 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142495 4517 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142499 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142503 4517 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:22.156933 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142506 4517 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142509 4517 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142511 4517 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142514 4517 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142517 4517 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142519 4517 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142522 4517 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142525 4517 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142528 4517 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142530 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142533 4517 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142536 4517 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142540 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142544 4517 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142549 4517 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142553 4517 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142557 4517 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142561 4517 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142566 4517 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142570 4517 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:22.157731 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142574 4517 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142581 4517 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142588 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142592 4517 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142597 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142602 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142606 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142610 4517 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142614 4517 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142618 4517 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142622 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142629 4517 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142635 4517 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142639 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142644 4517 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142648 4517 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142650 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142653 4517 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142656 4517 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:22.158399 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142660 4517 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142663 4517 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142666 4517 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142668 4517 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142671 4517 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142673 4517 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142676 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142678 4517 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.142729 4517 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142784 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142800 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142806 4517 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142811 4517 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142814 4517 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142819 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142823 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:22.159054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142831 4517 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142835 4517 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142839 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142844 4517 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142848 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142852 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142857 4517 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142862 4517 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142868 4517 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142873 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142878 4517 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142883 4517 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142887 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142892 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142897 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142901 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142906 4517 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142910 4517 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142914 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142919 4517 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142923 4517 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:22.159624 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142928 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142932 4517 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142936 4517 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142940 4517 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142949 4517 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142954 4517 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142958 4517 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142962 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142967 4517 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142973 4517 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142977 4517 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142984 4517 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142989 4517 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.142993 4517 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143004 4517 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143010 4517 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143014 4517 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143018 4517 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143023 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143027 4517 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:22.160334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143031 4517 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143035 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143039 4517 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143043 4517 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143046 4517 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143050 4517 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143054 4517 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143058 4517 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143063 4517 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143067 4517 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143071 4517 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143075 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143078 4517 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143081 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143083 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143087 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143089 4517 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143094 4517 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143097 4517 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:22.161008 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143099 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143105 4517 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143109 4517 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143112 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143117 4517 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143119 4517 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143122 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143125 4517 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143129 4517 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143133 4517 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143138 4517 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143141 4517 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143146 4517 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143150 4517 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143154 4517 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143158 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143161 4517 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143166 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143170 4517 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:22.161678 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143174 4517 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143177 4517 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143180 4517 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143182 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143184 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143187 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143193 4517 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143200 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143205 4517 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143210 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143214 4517 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143219 4517 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143225 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143230 4517 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143234 4517 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143239 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143243 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143248 4517 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143252 4517 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.143255 4517 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:22.162378 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.150775 4517 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.150807 4517 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.151016 4517 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.151034 4517 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151111 4517 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151117 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151121 4517 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151124 4517 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151127 4517 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151129 4517 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151132 4517 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151135 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151138 4517 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151141 4517 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151147 4517 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151153 4517 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151158 4517 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151162 4517 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151167 4517 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:22.163010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151171 4517 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151175 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151178 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151181 4517 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151183 4517 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151186 4517 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151188 4517 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151191 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151194 4517 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151197 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151201 4517 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151205 4517 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151209 4517 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151212 4517 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151215 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151220 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151225 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151229 4517 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151234 4517 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:22.163660 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151238 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151240 4517 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151243 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151246 4517 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151248 4517 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151251 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151254 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151257 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151259 4517 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151262 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151264 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151267 4517 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151270 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151272 4517 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151275 4517 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151278 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151280 4517 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151283 4517 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151285 4517 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151288 4517 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:22.164349 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151290 4517 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151293 4517 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151296 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151298 4517 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151301 4517 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151303 4517 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151306 4517 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151309 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151312 4517 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151315 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151317 4517 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151320 4517 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151322 4517 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151325 4517 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151328 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151330 4517 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151332 4517 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151335 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151337 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151340 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:22.165088 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151342 4517 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151345 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151347 4517 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151349 4517 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151352 4517 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151354 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151357 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151359 4517 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151362 4517 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151364 4517 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151367 4517 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151369 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151372 4517 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151375 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151377 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151380 4517 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151382 4517 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151385 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151387 4517 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151390 4517 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:22.165741 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151393 4517 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151396 4517 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151399 4517 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151402 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151405 4517 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151407 4517 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151410 4517 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151413 4517 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151415 4517 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151418 4517 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151423 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151427 4517 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.151457 4517 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151636 4517 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151642 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151645 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:22.166430 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151648 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151651 4517 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151655 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151658 4517 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151661 4517 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151665 4517 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151669 4517 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151673 4517 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151676 4517 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151680 4517 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151683 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151686 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151688 4517 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151691 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151693 4517 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151696 4517 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151700 4517 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151724 4517 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151729 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151733 4517 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:22.166982 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151738 4517 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151742 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151746 4517 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151751 4517 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151756 4517 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151761 4517 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151765 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151770 4517 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151773 4517 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151775 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151779 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151782 4517 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151784 4517 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151787 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151789 4517 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151794 4517 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151797 4517 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151800 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151802 4517 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151805 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:22.167652 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151807 4517 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151810 4517 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151813 4517 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151816 4517 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151819 4517 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151821 4517 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151824 4517 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151827 4517 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151830 4517 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151833 4517 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151835 4517 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151838 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151842 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151844 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151847 4517 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151849 4517 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151852 4517 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151854 4517 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151857 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151859 4517 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:22.168424 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151862 4517 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151865 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151867 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151870 4517 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151872 4517 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151876 4517 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151880 4517 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151883 4517 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151886 4517 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151889 4517 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151893 4517 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151897 4517 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151903 4517 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151907 4517 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151910 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151912 4517 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151915 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151918 4517 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:22.169105 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151920 4517 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151923 4517 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151926 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151928 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151931 4517 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151933 4517 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151936 4517 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151938 4517 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151941 4517 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151944 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151946 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151949 4517 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151952 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151955 4517 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151958 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151963 4517 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151967 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151971 4517 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151975 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151979 4517 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:22.169733 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151983 4517 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151988 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151992 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.151998 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152003 4517 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.152042 4517 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152083 4517 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152087 4517 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152090 4517 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152093 4517 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152096 4517 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152099 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152102 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152104 4517 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152107 4517 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152109 4517 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:22.170395 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152112 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152114 4517 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152117 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152119 4517 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152122 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152125 4517 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152127 4517 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152130 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152132 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152134 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152137 4517 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152139 4517 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152142 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152144 4517 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152147 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152149 4517 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152152 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152154 4517 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152156 4517 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:22.170957 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152159 4517 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152162 4517 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152165 4517 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152171 4517 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152176 4517 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152179 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152183 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152185 4517 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152188 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152191 4517 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152193 4517 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152196 4517 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152199 4517 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152201 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152206 4517 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152211 4517 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152215 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152219 4517 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152224 4517 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152228 4517 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:22.171613 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152233 4517 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152238 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152243 4517 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152247 4517 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152252 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152257 4517 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152260 4517 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152262 4517 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152265 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152267 4517 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152270 4517 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152273 4517 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152275 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152278 4517 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152281 4517 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152285 4517 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152288 4517 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152291 4517 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152294 4517 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152296 4517 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:22.172288 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152299 4517 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152301 4517 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152304 4517 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152307 4517 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152309 4517 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152311 4517 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152316 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152318 4517 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152321 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152324 4517 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152327 4517 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152329 4517 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152331 4517 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152334 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152336 4517 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152338 4517 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152341 4517 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152343 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152346 4517 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:22.173018 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152348 4517 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152352 4517 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152357 4517 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152359 4517 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152362 4517 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152365 4517 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152367 4517 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152370 4517 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152376 4517 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152379 4517 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152382 4517 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152385 4517 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152388 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152391 4517 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152393 4517 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152396 4517 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152398 4517 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:22.152401 4517 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:22.173645 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.152464 4517 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.153593 4517 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.153654 4517 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.153873 4517 server.go:1014] "Starting client certificate rotation" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.154009 4517 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.154069 4517 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:28:41 +0000 UTC" deadline="2028-05-22 17:38:36.800613042 +0000 UTC" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.154103 4517 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13697h4m14.646512854s" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.154410 4517 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.155751 4517 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.157552 4517 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.159319 4517 log.go:25] "Validated CRI v1 image API" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.160171 4517 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:34:22.174267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.161952 4517 fs.go:135] Filesystem UUIDs: map[73615d2a-4279-4ca9-9f39-4f9c936c230d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e97e6e66-acd8-421c-a221-6ea22b52c37d:/dev/nvme0n1p3] Oct 30 00:34:22.174688 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.161982 4517 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ef8519f2f57921e78d0cbc43a84adcf78d125514cecde68e63f5373a96592cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ef8519f2f57921e78d0cbc43a84adcf78d125514cecde68e63f5373a96592cb/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f2055fed2cb973618817524e7254b82e8f0c51e45dc2647e342bcffcfed7b82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f2055fed2cb973618817524e7254b82e8f0c51e45dc2647e342bcffcfed7b82/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/206cfd82e95202a2549277a2ac890c7a953febcad2aefe0736d466cfaf839611/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/206cfd82e95202a2549277a2ac890c7a953febcad2aefe0736d466cfaf839611/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fd39b3321ac2a3cb53f2481875fb203d4f0129e37582d94910d6ad339881821/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fd39b3321ac2a3cb53f2481875fb203d4f0129e37582d94910d6ad339881821/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43760e3abdb60e7366ff855e69ec0a5fc1fab5633b4cca56d55ff465a1aebfb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43760e3abdb60e7366ff855e69ec0a5fc1fab5633b4cca56d55ff465a1aebfb0/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/585e6d02764a61f1550beda8a8c4ce2b3d98a82f42e071b49cf16d8a001fff64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/585e6d02764a61f1550beda8a8c4ce2b3d98a82f42e071b49cf16d8a001fff64/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75b04f1a82854b483f4ade164edb481a99811cb1f85b1002ef1c3d6097410cde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75b04f1a82854b483f4ade164edb481a99811cb1f85b1002ef1c3d6097410cde/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81ce993d82a5521103059e452631659e6719d192c30a120e60cfd25b30347ffe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81ce993d82a5521103059e452631659e6719d192c30a120e60cfd25b30347ffe/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cf5dfebd102a6423e17241ba07ad036ecb8f1feaa362239b6a5ccaec78763e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cf5dfebd102a6423e17241ba07ad036ecb8f1feaa362239b6a5ccaec78763e4/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b17adb0349ceea2e111330a8a2f949c63fabfdbe0498e869c87baa71c97e1b12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b17adb0349ceea2e111330a8a2f949c63fabfdbe0498e869c87baa71c97e1b12/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b274b478ad62a0b147aa9e6c1411e7639312424d0a4fad4b3bc5b90880edb337/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b274b478ad62a0b147aa9e6c1411e7639312424d0a4fad4b3bc5b90880edb337/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc5b6a01276734b88c6053bdf78ed59fe8f43f248d94c240fd0d79225847bcf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc5b6a01276734b88c6053bdf78ed59fe8f43f248d94c240fd0d79225847bcf2/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de6481df9297cfdd627c26a66311bff025dcd73d2fb6219d3670eb8196b3a146/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de6481df9297cfdd627c26a66311bff025dcd73d2fb6219d3670eb8196b3a146/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/def8ab59d7a2782db2cda69fd22d8fa59a7f2371af36398380fef28a6b345bda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/def8ab59d7a2782db2cda69fd22d8fa59a7f2371af36398380fef28a6b345bda/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3e922f85fb4cd2b947e5c3cbf067473d4f7a20cd2d29d7421c6d37c22cdc7e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3e922f85fb4cd2b947e5c3cbf067473d4f7a20cd2d29d7421c6d37c22cdc7e1/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5b85cc09e62a99dc5348a46cd9498ad8bb1ad820458fb599f43f71cb4434f1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5b85cc09e62a99dc5348a46cd9498ad8bb1ad820458fb599f43f71cb4434f1c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/041a2fd8-4580-4074-a167-7cb444d727e0/volumes/kubernetes.io~projected/kube-api-access-nmr64:{mountpoint:/var/lib/kubelet/pods/041a2fd8-4580-4074-a167-7cb444d727e0/volumes/kubernetes.io~projected/kube-api-access-nmr64 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/041a2fd8-4580-4074-a167-7cb444d727e0/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/041a2fd8-4580-4074-a167-7cb444d727e0/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16d03985-22a9-42a8-a195-638cb40e9e2c/volumes/kubernetes.io~projected/kube-api-access-h72b5:{mountpoint:/var/lib/kubelet/pods/16d03985-22a9-42a8-a195-638cb40e9e2c/volumes/kubernetes.io~projected/kube-api-access-h72b5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9/volumes/kubernetes.io~projected/kube-api-access-njq4r:{mountpoint:/var/lib/kubelet/pods/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9/volumes/kubernetes.io~projected/kube-api-access-njq4r major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb/volumes/kubernetes.io~projected/kube-api-access-gbxzz:{mountpoint:/var/lib/kubelet/pods/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb/volumes/kubernetes.io~projected/kube-api-access-gbxzz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b703ac5-d445-4378-8d48-eea0ec2563fc/volumes/kubernetes.io~projected/kube-api-access-tkcpc:{mountpoint:/var/lib/kubelet/pods/3b703ac5-d445-4378-8d48-eea0ec2563fc/volumes/kubernetes.io~projected/kube-api-access-tkcpc major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b703ac5-d445-4378-8d48-eea0ec2563fc/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3b703ac5-d445-4378-8d48-eea0ec2563fc/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4667c989-ad40-4b50-9d15-c341751fff71/volumes/kubernetes.io~projected/kube-api-access-7wxgk:{mountpoint:/var/lib/kubelet/pods/4667c989-ad40-4b50-9d15-c341751fff71/volumes/kubernetes.io~projected/kube-api-access-7wxgk major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d0d2d6e-c723-4393-92e5-65c3e8b4f713/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4d0d2d6e-c723-4393-92e5-65c3e8b4f713/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61e5f3a5-6460-4b3e-a423-70d0000dd706/volumes/kubernetes.io~projected/kube-api-access-xvpqf:{mountpoint:/var/lib/kubelet/pods/61e5f3a5-6460-4b3e-a423-70d0000dd706/volumes/kubernetes.io~projected/kube-api-access-xvpqf major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68188523-c3a4-4a55-90a8-0ba680aad7e2/volumes/kubernetes.io~projected/kube-api-access-jb2dn:{mountpoint:/var/lib/kubelet/pods/68188523-c3a4-4a55-90a8-0ba680aad7e2/volumes/kubernetes.io~projected/kube-api-access-jb2dn major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72e30b08-acdb-4c39-9689-018566263138/volumes/kubernetes.io~projected/kube-api-access-lzh7g:{mountpoint:/var/lib/kubelet/pods/72e30b08-acdb-4c39-9689-018566263138/volumes/kubernetes.io~projected/kube-api-access-lzh7g major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75bd79f7-bef7-44d5-9257-8048cc570d2c/volumes/kubernetes.io~projected/kube-api-access-cf9b5:{mountpoint:/var/lib/kubelet/pods/75bd79f7-bef7-44d5-9257-8048cc570d2c/volumes/kubernetes.io~projected/kube-api-access-cf9b5 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e/volumes/kubernetes.io~projected/kube-api-access-pw7zp:{mountpoint:/var/lib/kubelet/pods/87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e/volumes/kubernetes.io~projected/kube-api-access-pw7zp major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b43fb340-5191-4f45-b6d9-9916afc9a2fd/volumes/kubernetes.io~projected/kube-api-access-6gkdb:{mountpoint:/var/lib/kubelet/pods/b43fb340-5191-4f45-b6d9-9916afc9a2fd/volumes/kubernetes.io~projected/kube-api-access-6gkdb major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0349f41-063d-4636-963b-4b88924b3670/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c0349f41-063d-4636-963b-4b88924b3670/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0349f41-063d-4636-963b-4b88924b3670/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c0349f41-063d-4636-963b-4b88924b3670/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0349f41-063d-4636-963b-4b88924b3670/volumes/kubernetes.io~projected/kube-api-access-fjvgx:{mountpoint:/var/lib/kubelet/pods/c0349f41-063d-4636-963b-4b88924b3670/volumes/kubernetes.io~projected/kube-api-access-fjvgx major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1d6ac87-947e-44dd-9a24-1b1df8755f06/volumes/kubernetes.io~projected/kube-api-access-4cbh9:{mountpoint:/var/lib/kubelet/pods/d1d6ac87-947e-44dd-9a24-1b1df8755f06/volumes/kubernetes.io~projected/kube-api-access-4cbh9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e868adcd-9f67-4757-b2e1-9cb31a968d02/volumes/kubernetes.io~projected/kube-api-access-4nfmc:{mountpoint:/var/lib/kubelet/pods/e868adcd-9f67-4757-b2e1-9cb31a968d02/volumes/kubernetes.io~projected/kube-api-access-4nfmc major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9/volumes/kubernetes.io~projected/kube-api-access-t4lbj:{mountpoint:/var/lib/kubelet/pods/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9/volumes/kubernetes.io~projected/kube-api-access-t4lbj major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/67a39a2890393f22863098cf3ebf086d12b070fba275c5e9595475ec6eedc47a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/41495c02cddc04e353927da136124680103c4537bee84ae3389c67fb47dec4ba/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cd665fe756655b3a277019135520a475c1072f11fef24584cb7221b48d6bd26a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/424e7ebfa82d0268ddecb0a675678274ac492eb6ccfd081c9609ff52ea5e4c5d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a29d19e4a742d5b63675d976845be74c7d837c52b8edc86fd3a4c8684dacfbf1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5c8c5201caaec7f0d63be6ed4b72fb2f32f99a521fd6bd4b6ba4c788c3f46346/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/18bacb3875a16c54f0fcb648e37b6eede3ba7059e07d3cc3da204ca2a28b6229/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7262d52ccd8b706b08dc4acfaa6399df4b002887097147fba0f0cb4df1ba8b86/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/59c6a78c67f716971b5f9cf25db998da6fcdae010dec1476c6ac2f09553998e3/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/048689cd70734336ea52a47ae4fe5a7b00944cf0d408bd9508ce7cd8771f80a9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2d3c903a23c40841fb6fdf972d9922963885f117e75d28e760203ff0158b49a4/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/e893a4e6bc8e865a5f8ec9d4888144c25af4d61b96522d2e817939129d0e8cc5/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4537c87e60059f692552b3dd3e3e84911f8c6cd5a01a853e59c7888bb1d36374/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/bbe71fc0a9dc30294d6713c7f3dc2d83c9c440fa0ce797f456ec5cdcbf9f633e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/31e6b08192c627a2fabbc6ad6bb7183970c6890859893858b49117401c9863bc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/fb8e75ae383ddad2e9586ca4894d0618fd9cd7fc814f6d09564e294473f0a967/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b51eebe32303afe791cc9cc95ae2aec486ce6ea1337b2854e41cc29734f7a28c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/containers/storage/overlay/289949819e6d80e2a59d0458a98682e302de7d2350ceea72a4c2c473af8a3e33/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/8a2b127dcacb2601c45a310d683bf754ea1dc2f7b059223a7966e98800c273dc/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e12234e89c569a394695b241f15b40cfecb1f5318d6e45b7adb5966b2ebbd413/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/03808ba1619332e3a39ae6110fe1b02ce1436dc28d682b7cc0ffea3b4d4a3354/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/2340e56d9a1c0a26f576cc02f4501097b910004eeba30c01d0cdadf6e74bc986/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/5c68ff52347d1ee2cbf4ede003c6648988c0b0c570e65b10811622b5740ff530/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/e709ebca44010bc75ee073302c058c2d9d64e91eecc13b0b7446f4ecc0861cd0/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/a73c92c70681f19f41c0a3007467c5ac0931d302eefc84ededfa343e2a8f688d/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/2c342c0ee3dadaf9c93727139ba61b2630edacf72625fcd78e23f107aac2a4c9/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/8c9173d592279f112ca025f523da760138ca796c1b2386587b41a7ab0e0bfda1/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/17b667116bedf226df19d3fc420c67bc8c76fee53158975d398059c337f2f15f/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/87227476eece19f78d9430dbd0cb62839e569f678f6b2b1a60088bbcde1cb7af/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/76f522b2e573d35279805bde9bdcf72bd1bf79b00591f0ea62b915ad27a2c2f7/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/e78c020fd6eb6350e7e35a3767184fd4a60066f8f8265c8ef476393681ef8371/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/d3c6f4cb5fc636ae3132e3e211674f5a294b2c50cbc5826a1f17c5c72e19bcbf/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/0e9a69ba12315650a957cbb04abd434a9447f84579da0dfd491b7308cc9eae59/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e478a0dcf532caf157a7d61589a8373af726e8916595c4402b54e687f97eab77/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9f3a6826647105103b7120ff9032460f8682aca6a838f0904d7734a48feb9ab4/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f8289f1d5061973c1b1b847f9d73f277d898d993b22056d0095cd4b78b2c91a8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/451f4b11fca5d895d4abb87356d184911695749bd65a1f1c434ab575974afeac/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0e13b36c9de7ca5b9d47b27567eb54a7537ac79c9e190f0bd15b0ec5bf20735a/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/598ba0184902010fccd81fbcdd439422c1577684ef23fbfed86a4862005ae180/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8fcb86708a0e9671983aa38f0562b5dffe7c72c7c268778e4df3db8b74ec1534/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/50ca99a20020871199d12d71a9b575525513c10f4737e7b28d39c8eb8730286b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/815db562369779d5a5266e43d63993eef1ece0277713b8cf7ccd80331c54d155/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.170267 4517 manager.go:217] Machine: {Timestamp:2026-10-30 00:34:22.169381451 +0000 UTC m=+0.081508622 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100401 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22a6cfa14ea2d2c30f5f19a2cae160 SystemUUID:ec22a6cf-a14e-a2d2-c30f-5f19a2cae160 BootID:99efdd17-83eb-4816-b4de-c8be21e66253 Filesystems:[{Device:/var/lib/kubelet/pods/b43fb340-5191-4f45-b6d9-9916afc9a2fd/volumes/kubernetes.io~projected/kube-api-access-6gkdb DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d0d2d6e-c723-4393-92e5-65c3e8b4f713/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75b04f1a82854b483f4ade164edb481a99811cb1f85b1002ef1c3d6097410cde/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/041a2fd8-4580-4074-a167-7cb444d727e0/volumes/kubernetes.io~projected/kube-api-access-nmr64 DeviceMajor:0 DeviceMinor:234 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/75bd79f7-bef7-44d5-9257-8048cc570d2c/volumes/kubernetes.io~projected/kube-api-access-cf9b5 DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f2055fed2cb973618817524e7254b82e8f0c51e45dc2647e342bcffcfed7b82/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/206cfd82e95202a2549277a2ac890c7a953febcad2aefe0736d466cfaf839611/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4667c989-ad40-4b50-9d15-c341751fff71/volumes/kubernetes.io~projected/kube-api-access-7wxgk DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72e30b08-acdb-4c39-9689-018566263138/volumes/kubernetes.io~projected/kube-api-access-lzh7g DeviceMajor:0 DeviceMinor:226 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9/volumes/kubernetes.io~projected/kube-api-access-t4lbj DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-178 DeviceMajor:0 DeviceMinor:178 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/def8ab59d7a2782db2cda69fd22d8fa59a7f2371af36398380fef28a6b345bda/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0349f41-063d-4636-963b-4b88924b3670/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ef8519f2f57921e78d0cbc43a84adcf78d125514cecde68e63f5373a96592cb/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e/volumes/kubernetes.io~projected/kube-api-access-pw7zp DeviceMajor:0 DeviceMinor:222 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1d6ac87-947e-44dd-9a24-1b1df8755f06/volumes/kubernetes.io~projected/kube-api-access-4cbh9 DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/585e6d02764a61f1550beda8a8c4ce2b3d98a82f42e071b49cf16d8a001fff64/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b703ac5-d445-4378-8d48-eea0ec2563fc/volumes/kubernetes.io~projected/kube-api-access-tkcpc DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e868adcd-9f67-4757-b2e1-9cb31a968d02/volumes/kubernetes.io~projected/kube-api-access-4nfmc DeviceMajor:0 DeviceMinor:213 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3e922f85fb4cd2b947e5c3cbf067473d4f7a20cd2d29d7421c6d37c22cdc7e1/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b703ac5-d445-4378-8d48-eea0ec2563fc/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/68188523-c3a4-4a55-90a8-0ba680aad7e2/volumes/kubernetes.io~projected/kube-api-access-jb2dn DeviceMajor:0 DeviceMinor:225 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc5b6a01276734b88c6053bdf78ed59fe8f43f248d94c240fd0d79225847bcf2/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b17adb0349ceea2e111330a8a2f949c63fabfdbe0498e869c87baa71c97e1b12/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/041a2fd8-4580-4074-a167-7cb444d727e0/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/16d03985-22a9-42a8-a195-638cb40e9e2c/volumes/kubernetes.io~projected/kube-api-access-h72b5 DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43760e3abdb60e7366ff855e69ec0a5fc1fab5633b4cca56d55ff465a1aebfb0/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b274b478ad62a0b147aa9e6c1411e7639312424d0a4fad4b3bc5b90880edb337/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c0349f41-063d-4636-963b-4b88924b3670/volumes/kubernetes.io~projected/kube-api-access-fjvgx DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81ce993d82a5521103059e452631659e6719d192c30a120e60cfd25b30347ffe/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de6481df9297cfdd627c26a66311bff025dcd73d2fb6219d3670eb8196b3a146/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cf5dfebd102a6423e17241ba07ad036ecb8f1feaa362239b6a5ccaec78763e4/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb/volumes/kubernetes.io~projected/kube-api-access-gbxzz DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9/volumes/kubernetes.io~projected/kube-api-access-njq4r DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fd39b3321ac2a3cb53f2481875fb203d4f0129e37582d94910d6ad339881821/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0349f41-063d-4636-963b-4b88924b3670/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/61e5f3a5-6460-4b3e-a423-70d0000dd706/volumes/kubernetes.io~projected/kube-api-access-xvpqf DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5b85cc09e62a99dc5348a46cd9498ad8bb1ad820458fb599f43f71cb4434f1c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:81ce993d82a5521 MacAddress:66:d0:fa:47:30:8f Speed:10000 Mtu:8901} {Name:b274b478ad62a0b MacAddress:aa:cc:1a:f3:63:da Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e7:e1:f3:6d Speed:0 Mtu:9001} {Name:br-int MacAddress:02:ed:6b:2e:45:58 Speed:0 Mtu:8901} {Name:cc5b6a01276734b MacAddress:a2:60:51:b9:18:3b Speed:10000 Mtu:8901} {Name:de6481df9297cfd MacAddress:b2:90:c1:1c:4d:98 Speed:10000 Mtu:8901} {Name:def8ab59d7a2782 MacAddress:96:21:73:4f:80:2d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e7:e1:f3:6d Speed:0 Mtu:9001} {Name:f3e922f85fb4cd2 MacAddress:d6:2c:bd:9f:58:76 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f6:48:db:dc:87:aa Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5a:e7:25:8a:7a:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.171181 4517 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.171258 4517 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.171885 4517 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.171923 4517 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.172356 4517 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.172370 4517 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.172390 4517 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.172427 4517 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.172542 4517 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.172838 4517 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.173038 4517 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.173098 4517 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.173119 4517 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.173135 4517 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.173150 4517 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.173163 4517 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.174594 4517 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:34:22.174905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.174612 4517 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:34:22.175760 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.175520 4517 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:34:22.186216 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186195 4517 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:34:22.186216 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186212 4517 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:34:22.186425 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186335 4517 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 00:34:22.186581 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186567 4517 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:34:22.186638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186598 4517 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:34:22.186780 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186755 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:34:22.186780 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186777 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186784 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186790 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186796 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186802 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186808 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186814 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186825 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186832 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186841 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186851 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:34:22.186880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186881 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:34:22.187297 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.186891 4517 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:34:22.191188 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.191155 4517 server.go:1263] "Started kubelet" Oct 30 00:34:22.191522 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.191455 4517 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:34:22.191665 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.191548 4517 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:34:22.191823 ip-10-0-132-84 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:34:22.191982 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.191837 4517 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:34:22.192307 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.192290 4517 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:34:22.194265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.194249 4517 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:34:22.195881 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.195571 4517 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:34:22.195881 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.195856 4517 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.196619 4517 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:28:43 +0000 UTC" deadline="2028-04-29 11:03:13.582338114 +0000 UTC" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.196661 4517 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13138h28m51.385679821s" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:22.196694 4517 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-84.ec2.internal\" not found" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.196790 4517 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.196798 4517 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.197003 4517 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198592 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9" volumeName="kubernetes.io/projected/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-kube-api-access-njq4r" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198627 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b47aade-5b0f-4ab4-ac11-e71acc17a9cb" volumeName="kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-script-lib" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198643 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68188523-c3a4-4a55-90a8-0ba680aad7e2" volumeName="kubernetes.io/projected/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-api-access-jb2dn" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198656 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72e30b08-acdb-4c39-9689-018566263138" volumeName="kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198670 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1d6ac87-947e-44dd-9a24-1b1df8755f06" volumeName="kubernetes.io/projected/d1d6ac87-947e-44dd-9a24-1b1df8755f06-kube-api-access-4cbh9" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198687 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68188523-c3a4-4a55-90a8-0ba680aad7e2" volumeName="kubernetes.io/empty-dir/68188523-c3a4-4a55-90a8-0ba680aad7e2-data-volume" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198698 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b43fb340-5191-4f45-b6d9-9916afc9a2fd" volumeName="kubernetes.io/empty-dir/b43fb340-5191-4f45-b6d9-9916afc9a2fd-tmp-dir" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198731 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9" volumeName="kubernetes.io/configmap/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-serviceca" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198745 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b47aade-5b0f-4ab4-ac11-e71acc17a9cb" volumeName="kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-env-overrides" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198757 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b47aade-5b0f-4ab4-ac11-e71acc17a9cb" volumeName="kubernetes.io/projected/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-kube-api-access-gbxzz" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198770 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d0d2d6e-c723-4393-92e5-65c3e8b4f713" volumeName="kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198782 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75bd79f7-bef7-44d5-9257-8048cc570d2c" volumeName="kubernetes.io/configmap/75bd79f7-bef7-44d5-9257-8048cc570d2c-iptables-alerter-script" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198796 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="041a2fd8-4580-4074-a167-7cb444d727e0" volumeName="kubernetes.io/projected/041a2fd8-4580-4074-a167-7cb444d727e0-kube-api-access-nmr64" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198808 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="041a2fd8-4580-4074-a167-7cb444d727e0" volumeName="kubernetes.io/secret/041a2fd8-4580-4074-a167-7cb444d727e0-signing-key" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198830 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72e30b08-acdb-4c39-9689-018566263138" volumeName="kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198842 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e" volumeName="kubernetes.io/projected/87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e-kube-api-access-pw7zp" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198856 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" volumeName="kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198871 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16d03985-22a9-42a8-a195-638cb40e9e2c" volumeName="kubernetes.io/empty-dir/16d03985-22a9-42a8-a195-638cb40e9e2c-tmp-dir" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198884 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b703ac5-d445-4378-8d48-eea0ec2563fc" volumeName="kubernetes.io/secret/3b703ac5-d445-4378-8d48-eea0ec2563fc-cert" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198898 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b43fb340-5191-4f45-b6d9-9916afc9a2fd" volumeName="kubernetes.io/configmap/b43fb340-5191-4f45-b6d9-9916afc9a2fd-config-volume" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198909 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0349f41-063d-4636-963b-4b88924b3670" volumeName="kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-tmp" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198921 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1d6ac87-947e-44dd-9a24-1b1df8755f06" volumeName="kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-daemon-config" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198935 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="041a2fd8-4580-4074-a167-7cb444d727e0" volumeName="kubernetes.io/configmap/041a2fd8-4580-4074-a167-7cb444d727e0-signing-cabundle" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198948 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16d03985-22a9-42a8-a195-638cb40e9e2c" volumeName="kubernetes.io/projected/16d03985-22a9-42a8-a195-638cb40e9e2c-kube-api-access-h72b5" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198960 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e7f74c1-965a-4ccc-a8f0-2dbd1770908e" volumeName="kubernetes.io/configmap/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-konnectivity-ca" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198972 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61e5f3a5-6460-4b3e-a423-70d0000dd706" volumeName="kubernetes.io/projected/61e5f3a5-6460-4b3e-a423-70d0000dd706-kube-api-access-xvpqf" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198984 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0349f41-063d-4636-963b-4b88924b3670" volumeName="kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-etc-tuned" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.198995 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0349f41-063d-4636-963b-4b88924b3670" volumeName="kubernetes.io/projected/c0349f41-063d-4636-963b-4b88924b3670-kube-api-access-fjvgx" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199022 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" volumeName="kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-binary-copy" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199035 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" volumeName="kubernetes.io/projected/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-kube-api-access-t4lbj" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199048 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b47aade-5b0f-4ab4-ac11-e71acc17a9cb" volumeName="kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-config" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199074 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b703ac5-d445-4378-8d48-eea0ec2563fc" volumeName="kubernetes.io/projected/3b703ac5-d445-4378-8d48-eea0ec2563fc-kube-api-access-tkcpc" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199089 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e7f74c1-965a-4ccc-a8f0-2dbd1770908e" volumeName="kubernetes.io/secret/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-agent-certs" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199101 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4667c989-ad40-4b50-9d15-c341751fff71" volumeName="kubernetes.io/projected/4667c989-ad40-4b50-9d15-c341751fff71-kube-api-access-7wxgk" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199122 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" volumeName="kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199137 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b47aade-5b0f-4ab4-ac11-e71acc17a9cb" volumeName="kubernetes.io/secret/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199148 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72e30b08-acdb-4c39-9689-018566263138" volumeName="kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199191 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72e30b08-acdb-4c39-9689-018566263138" volumeName="kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199205 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75bd79f7-bef7-44d5-9257-8048cc570d2c" volumeName="kubernetes.io/projected/75bd79f7-bef7-44d5-9257-8048cc570d2c-kube-api-access-cf9b5" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199222 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b43fb340-5191-4f45-b6d9-9916afc9a2fd" volumeName="kubernetes.io/projected/b43fb340-5191-4f45-b6d9-9916afc9a2fd-kube-api-access-6gkdb" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199239 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1d6ac87-947e-44dd-9a24-1b1df8755f06" volumeName="kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cni-binary-copy" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199255 4517 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e868adcd-9f67-4757-b2e1-9cb31a968d02" volumeName="kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc" seLinuxMountContext="" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199266 4517 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:34:22.200735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.199273 4517 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:34:22.208588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.208487 4517 factory.go:55] Registering systemd factory Oct 30 00:34:22.208588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.208515 4517 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:34:22.209473 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.209452 4517 factory.go:153] Registering CRI-O factory Oct 30 00:34:22.209568 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.209497 4517 factory.go:223] Registration of the crio container factory successfully Oct 30 00:34:22.209622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.209574 4517 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:34:22.209622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.209600 4517 factory.go:103] Registering Raw factory Oct 30 00:34:22.209622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.209616 4517 manager.go:1199] Started watching for new ooms in manager Oct 30 00:34:22.210115 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.210100 4517 manager.go:319] Starting recovery of all containers Oct 30 00:34:22.224946 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.224889 4517 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:34:22.226739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.226699 4517 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:34:22.226911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.226900 4517 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:34:22.227009 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.226992 4517 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:34:22.227433 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.227087 4517 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:34:22.227433 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:22.227121 4517 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 00:34:22.227433 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.227199 4517 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 00:34:22.228746 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.228701 4517 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:34:22.241466 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.241415 4517 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:34:22.249969 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.249902 4517 generic.go:356] "Generic (PLEG): container finished" podID="32199d49f1d2f12cbdbf36311255bf8c" containerID="39a119fdadc0a9c2b43c47719fc2d88a4123cc38b018e685cc7ce2fcc32c1e43" exitCode=0 Oct 30 00:34:22.251636 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.251616 4517 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:34:22.267492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.267419 4517 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="1d1f7dac4e7e8c54c6d860a50864e860b750baaf727d50b9ed24cd8e16da630c" exitCode=0 Oct 30 00:34:22.267492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.267478 4517 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="cd8726e8de5035d54f26d73e6afb7f356a21050ef61114d310cec00807b1f4bf" exitCode=0 Oct 30 00:34:22.267492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.267488 4517 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="326d7e663e9d48db865a527adc8d6db4d5a9508bbc1dfa517bd8ff020dbeae36" exitCode=0 Oct 30 00:34:22.267492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.267497 4517 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="7163be8b300e2d16f5e8ad84f7a52bb7568127e293c58d05794b1768b03e82eb" exitCode=0 Oct 30 00:34:22.267806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.267507 4517 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="f6194cc2f72607ebd8e2f6cd1fb9f9bcb03744784595a73fd41583c746022552" exitCode=0 Oct 30 00:34:22.267806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.267521 4517 generic.go:356] "Generic (PLEG): container finished" podID="f6dd358a-8c7b-4a22-8181-fa6b5aa931a9" containerID="aedcc55647313538671986be4d904a0b5f449e35bb0b2e2055c60be9dac18590" exitCode=0 Oct 30 00:34:22.300217 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.300183 4517 manager.go:324] Recovery completed Oct 30 00:34:22.323117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323084 4517 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:34:22.323117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323109 4517 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:34:22.323117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323130 4517 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:34:22.323345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323254 4517 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 00:34:22.323345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323265 4517 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 00:34:22.323345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323283 4517 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 00:34:22.323345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323288 4517 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 00:34:22.323345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323293 4517 policy_none.go:50] "Start" Oct 30 00:34:22.323345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323301 4517 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:34:22.323345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323309 4517 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:34:22.323696 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323450 4517 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 00:34:22.323696 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323462 4517 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 00:34:22.323696 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.323471 4517 policy_none.go:44] "Start" Oct 30 00:34:22.327521 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:22.327495 4517 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 00:34:22.329988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.329074 4517 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:34:22.336484 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.336458 4517 manager.go:341] "Starting Device Plugin manager" Oct 30 00:34:22.336609 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:22.336502 4517 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:34:22.336609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.336516 4517 server.go:87] "Starting device plugin registration server" Oct 30 00:34:22.336835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.336818 4517 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:34:22.336923 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.336839 4517 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:34:22.337084 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.337066 4517 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:34:22.337190 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.337159 4517 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:34:22.337244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.337203 4517 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:34:22.337244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.337226 4517 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:34:22.337383 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.337290 4517 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:34:22.337189055Z","UUID":"eac3dd6e-bc69-4ae3-9e5d-0326c5a759a4","Handler":null,"Name":"","Endpoint":""} Oct 30 00:34:22.338376 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:22.338350 4517 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:34:22.338556 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.338538 4517 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:34:22.338638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.338577 4517 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:34:22.437425 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.437391 4517 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:34:22.438680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.438656 4517 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:34:22.438807 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.438693 4517 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:34:22.438807 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.438727 4517 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:34:22.438807 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.438760 4517 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.472066 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.472035 4517 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.472181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.472114 4517 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.528071 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.528007 4517 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal"] Oct 30 00:34:22.528731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.528657 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" event={"ID":"32199d49f1d2f12cbdbf36311255bf8c","Type":"ContainerStarted","Data":"ca7f19dcc24517a5d8dcfc0c6b5be10e9937d6c846e44cc10860e65c73c6ce26"} Oct 30 00:34:22.528846 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.528742 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" event={"ID":"32199d49f1d2f12cbdbf36311255bf8c","Type":"ContainerDied","Data":"39a119fdadc0a9c2b43c47719fc2d88a4123cc38b018e685cc7ce2fcc32c1e43"} Oct 30 00:34:22.528846 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.528767 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" event={"ID":"32199d49f1d2f12cbdbf36311255bf8c","Type":"ContainerStarted","Data":"75b04f1a82854b483f4ade164edb481a99811cb1f85b1002ef1c3d6097410cde"} Oct 30 00:34:22.528846 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.528773 4517 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.528846 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.528784 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" event={"ID":"e7678942b8a9336cefbe126b25b5ab55","Type":"ContainerStarted","Data":"7ad366bc8c1ab8c0a53e6d2497d4f235b8d02f53a6b2127fb46afce64e0c99f9"} Oct 30 00:34:22.528846 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.528803 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" event={"ID":"e7678942b8a9336cefbe126b25b5ab55","Type":"ContainerStarted","Data":"3fd39b3321ac2a3cb53f2481875fb203d4f0129e37582d94910d6ad339881821"} Oct 30 00:34:22.529125 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.528997 4517 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.550150 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.550117 4517 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:34:22.550315 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:22.550174 4517 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-84.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.550554 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.550538 4517 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:34:22.550610 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:22.550580 4517 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.628926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.628892 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.629073 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.628925 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.629073 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.628965 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7678942b8a9336cefbe126b25b5ab55-config\") pod \"kube-apiserver-proxy-ip-10-0-132-84.ec2.internal\" (UID: \"e7678942b8a9336cefbe126b25b5ab55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.729314 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.729278 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.729314 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.729312 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.729545 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.729329 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7678942b8a9336cefbe126b25b5ab55-config\") pod \"kube-apiserver-proxy-ip-10-0-132-84.ec2.internal\" (UID: \"e7678942b8a9336cefbe126b25b5ab55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.729545 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.729370 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7678942b8a9336cefbe126b25b5ab55-config\") pod \"kube-apiserver-proxy-ip-10-0-132-84.ec2.internal\" (UID: \"e7678942b8a9336cefbe126b25b5ab55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.729545 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.729397 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:22.729545 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:22.729404 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32199d49f1d2f12cbdbf36311255bf8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" (UID: \"32199d49f1d2f12cbdbf36311255bf8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:23.175264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.175218 4517 apiserver.go:52] "Watching apiserver" Oct 30 00:34:23.210085 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.210052 4517 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:34:23.210510 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.210490 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal","openshift-multus/multus-additional-cni-plugins-zxlcf","openshift-dns/node-resolver-62mqm","openshift-insights/insights-runtime-extractor-lj5vr","openshift-network-operator/iptables-alerter-5whwd","kube-system/global-pull-secret-syncer-kprr7","openshift-ovn-kubernetes/ovnkube-node-qgg9k","kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh","openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7","openshift-service-ca/service-ca-5fffc97bdf-fqjft","openshift-cluster-node-tuning-operator/tuned-md6gg","openshift-dns/dns-default-tg442","openshift-network-diagnostics/network-check-target-6ncmx","openshift-image-registry/node-ca-zbzqk","openshift-multus/network-metrics-daemon-87vfm","openshift-ingress-canary/ingress-canary-qgjbc","kube-system/konnectivity-agent-jz5dq","openshift-insights/periodic-gathering-xtmdf-8fqw8","openshift-multus/multus-f92b8"] Oct 30 00:34:23.212679 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.212656 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:23.217885 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.217862 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.218252 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.218240 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.226176 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226142 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:34:23.226512 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226494 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:34:23.226690 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226671 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:34:23.226791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226696 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.226791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226756 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ltmbf\"" Oct 30 00:34:23.226908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226821 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.226908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226818 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x59g5\"" Oct 30 00:34:23.226908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226847 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s9968\"" Oct 30 00:34:23.226908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226857 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.227100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226990 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:34:23.227100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.226991 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.227100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.227020 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.227100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.227064 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.227100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.227068 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk5pr\"" Oct 30 00:34:23.227100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.227074 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.227100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.227082 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.235837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.235820 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.235975 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.235961 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.236194 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.235961 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:34:23.236194 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.235969 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qpxdw\"" Oct 30 00:34:23.236194 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.235987 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:34:23.236194 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.236015 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.236194 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.236032 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q972j\"" Oct 30 00:34:23.236614 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.236599 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.236819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.236799 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.236957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.236942 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.237027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.236966 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:34:23.237027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237018 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:34:23.237130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237030 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:34:23.237130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237043 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:34:23.237215 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237147 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:34:23.237244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237227 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.237305 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237292 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.237411 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237391 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:34:23.237504 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237491 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.237566 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237553 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 00:34:23.237621 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237568 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 00:34:23.237621 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237592 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b4svw\"" Oct 30 00:34:23.237621 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237605 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.237812 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237647 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qff2g\"" Oct 30 00:34:23.237812 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237683 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.237894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237860 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tcw96\"" Oct 30 00:34:23.237894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.237872 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:34:23.238068 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238026 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:34:23.238068 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238045 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.238068 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238051 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8qb6v\"" Oct 30 00:34:23.238068 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238058 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-75g57\"" Oct 30 00:34:23.238068 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238064 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:34:23.238332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238069 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 00:34:23.238332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238103 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-th8hp\"" Oct 30 00:34:23.238332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238137 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:34:23.238332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238265 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-smldl\"" Oct 30 00:34:23.238332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238295 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:34:23.238332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238327 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c889x\"" Oct 30 00:34:23.238617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238336 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2pzq\"" Oct 30 00:34:23.238617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238417 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:34:23.238617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238433 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:34:23.238617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238453 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pc46q\"" Oct 30 00:34:23.238617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238551 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78zxw\"" Oct 30 00:34:23.238617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.238584 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pkqjh\"" Oct 30 00:34:23.248562 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.248542 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.249084 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.249067 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 00:34:23.294973 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.294945 4517 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:34:23.295158 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.295075 4517 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:23.298144 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.298129 4517 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:34:23.316018 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.315988 4517 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:34:23.316149 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.316048 4517 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-84.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-84.ec2.internal" Oct 30 00:34:23.316569 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.316557 4517 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:34:23.316617 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.316589 4517 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-84.ec2.internal" Oct 30 00:34:23.326405 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.326372 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.326405 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.326386 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:34:23.326405 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.326384 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:34:23.326588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.326376 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:34:23.331629 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331604 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-host\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.331741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331644 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvgx\" (UniqueName: \"kubernetes.io/projected/c0349f41-063d-4636-963b-4b88924b3670-kube-api-access-fjvgx\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.331741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331666 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.331846 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331809 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-multus-certs\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.331891 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331856 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fb340-5191-4f45-b6d9-9916afc9a2fd-config-volume\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.331891 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331886 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-slash\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.331982 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331904 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-ovn\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.331982 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331958 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-netd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.331982 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331977 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-daemon-config\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.332129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.331993 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-etc-kubernetes\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.332129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332009 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h72b5\" (UniqueName: \"kubernetes.io/projected/16d03985-22a9-42a8-a195-638cb40e9e2c-kube-api-access-h72b5\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:34:23.332129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332052 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-socket-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.332129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332061 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fb340-5191-4f45-b6d9-9916afc9a2fd-config-volume\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.332129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332117 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-etc-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.332390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332158 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysconfig\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.332390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332186 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-system-cni-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.332390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332214 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzh7g\" (UniqueName: \"kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:23.332390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332268 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.332390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332371 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxgk\" (UniqueName: \"kubernetes.io/projected/4667c989-ad40-4b50-9d15-c341751fff71-kube-api-access-7wxgk\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.332622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332402 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-daemon-config\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.332622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332405 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njq4r\" (UniqueName: \"kubernetes.io/projected/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-kube-api-access-njq4r\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:34:23.332622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332455 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-kubelet\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.332622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332483 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-script-lib\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.332622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332558 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:23.332622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332586 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbh9\" (UniqueName: \"kubernetes.io/projected/d1d6ac87-947e-44dd-9a24-1b1df8755f06-kube-api-access-4cbh9\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.332622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332604 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-sys-fs\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.332932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332641 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.332932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332692 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxzz\" (UniqueName: \"kubernetes.io/projected/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-kube-api-access-gbxzz\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.332932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332696 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-script-lib\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.332932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332792 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-systemd\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.332932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332844 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-tmp\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.332932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332872 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.332932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332922 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332969 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cni-binary-copy\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.332968 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-tmp\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333002 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333033 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333081 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68188523-c3a4-4a55-90a8-0ba680aad7e2-crio-socket\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333106 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-conf\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333130 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-registration-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333138 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cni-binary-copy\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333171 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333177 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpqf\" (UniqueName: \"kubernetes.io/projected/61e5f3a5-6460-4b3e-a423-70d0000dd706-kube-api-access-xvpqf\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:23.333238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333212 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333234 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-bin\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333262 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333281 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-agent-certs\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333296 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-etc-tuned\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333342 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333366 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0349f41-063d-4636-963b-4b88924b3670-etc-tuned\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333401 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-dbus\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333466 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-k8s-cni-cncf-io\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333492 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-multus\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333512 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-hostroot\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333536 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2dn\" (UniqueName: \"kubernetes.io/projected/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-api-access-jb2dn\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333555 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333582 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/041a2fd8-4580-4074-a167-7cb444d727e0-signing-cabundle\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333674 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-system-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333681 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-agent-certs\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333745 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-device-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333783 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9b5\" (UniqueName: \"kubernetes.io/projected/75bd79f7-bef7-44d5-9257-8048cc570d2c-kube-api-access-cf9b5\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333846 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr64\" (UniqueName: \"kubernetes.io/projected/041a2fd8-4580-4074-a167-7cb444d727e0-kube-api-access-nmr64\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333883 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-host\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333888 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/041a2fd8-4580-4074-a167-7cb444d727e0-signing-cabundle\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.333923 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-systemd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334003 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-var-lib-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334037 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334054 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-bin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334071 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-serviceca\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334088 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-node-log\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334102 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-log-socket\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334117 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-kubernetes\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334138 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:23.334241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334177 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d03985-22a9-42a8-a195-638cb40e9e2c-hosts-file\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334275 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334321 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68188523-c3a4-4a55-90a8-0ba680aad7e2-data-volume\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334333 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-serviceca\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334342 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-config\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334379 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-sys\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334417 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68188523-c3a4-4a55-90a8-0ba680aad7e2-data-volume\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334426 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-lib-modules\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334458 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334502 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-kubelet\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334547 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75bd79f7-bef7-44d5-9257-8048cc570d2c-iptables-alerter-script\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334562 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovnkube-config\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334597 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkdb\" (UniqueName: \"kubernetes.io/projected/b43fb340-5191-4f45-b6d9-9916afc9a2fd-kube-api-access-6gkdb\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334691 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334733 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-original-pull-secret\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334840 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75bd79f7-bef7-44d5-9257-8048cc570d2c-iptables-alerter-script\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334847 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-konnectivity-ca\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334889 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334908 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334917 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cnibin\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.334951 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-netns\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335018 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43fb340-5191-4f45-b6d9-9916afc9a2fd-tmp-dir\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335035 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f74c1-965a-4ccc-a8f0-2dbd1770908e-konnectivity-ca\") pod \"konnectivity-agent-jz5dq\" (UID: \"3e7f74c1-965a-4ccc-a8f0-2dbd1770908e\") " pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:34:23.335035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335046 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b703ac5-d445-4378-8d48-eea0ec2563fc-cert\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335069 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335094 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75bd79f7-bef7-44d5-9257-8048cc570d2c-host-slash\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335155 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335159 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335183 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43fb340-5191-4f45-b6d9-9916afc9a2fd-tmp-dir\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335235 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-systemd-units\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335271 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335289 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b703ac5-d445-4378-8d48-eea0ec2563fc-cert\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335321 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lbj\" (UniqueName: \"kubernetes.io/projected/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-kube-api-access-t4lbj\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335414 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335435 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-netns\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335436 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335449 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-run\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335485 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-kubelet-config\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335524 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkcpc\" (UniqueName: \"kubernetes.io/projected/3b703ac5-d445-4378-8d48-eea0ec2563fc-kube-api-access-tkcpc\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335556 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-socket-dir-parent\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335575 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/041a2fd8-4580-4074-a167-7cb444d727e0-signing-key\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335591 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-env-overrides\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335606 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-modprobe-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335659 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-var-lib-kubelet\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335733 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cnibin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335789 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-env-overrides\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335834 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-conf-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335879 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/041a2fd8-4580-4074-a167-7cb444d727e0-signing-key\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335887 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d03985-22a9-42a8-a195-638cb40e9e2c-tmp-dir\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:34:23.335933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335915 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:23.336726 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335976 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.336726 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.335984 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d03985-22a9-42a8-a195-638cb40e9e2c-tmp-dir\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:34:23.336726 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.336076 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7zp\" (UniqueName: \"kubernetes.io/projected/87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e-kube-api-access-pw7zp\") pod \"migrator-66df5d496b-pg7p7\" (UID: \"87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" Oct 30 00:34:23.350911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.350884 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvgx\" (UniqueName: \"kubernetes.io/projected/c0349f41-063d-4636-963b-4b88924b3670-kube-api-access-fjvgx\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.351825 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.351794 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzh7g\" (UniqueName: \"kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g\") pod \"periodic-gathering-xtmdf-8fqw8\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:34:23.351918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.351885 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njq4r\" (UniqueName: \"kubernetes.io/projected/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-kube-api-access-njq4r\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:34:23.351954 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.351887 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpqf\" (UniqueName: \"kubernetes.io/projected/61e5f3a5-6460-4b3e-a423-70d0000dd706-kube-api-access-xvpqf\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:23.352121 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.352097 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbh9\" (UniqueName: \"kubernetes.io/projected/d1d6ac87-947e-44dd-9a24-1b1df8755f06-kube-api-access-4cbh9\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.352265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.352251 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxzz\" (UniqueName: \"kubernetes.io/projected/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-kube-api-access-gbxzz\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.352371 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.352353 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2dn\" (UniqueName: \"kubernetes.io/projected/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-api-access-jb2dn\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.352819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.352805 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9b5\" (UniqueName: \"kubernetes.io/projected/75bd79f7-bef7-44d5-9257-8048cc570d2c-kube-api-access-cf9b5\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:34:23.353641 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.353627 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkdb\" (UniqueName: \"kubernetes.io/projected/b43fb340-5191-4f45-b6d9-9916afc9a2fd-kube-api-access-6gkdb\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.353934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.353919 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmr64\" (UniqueName: \"kubernetes.io/projected/041a2fd8-4580-4074-a167-7cb444d727e0-kube-api-access-nmr64\") pod \"service-ca-5fffc97bdf-fqjft\" (UID: \"041a2fd8-4580-4074-a167-7cb444d727e0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" Oct 30 00:34:23.354836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.354821 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkcpc\" (UniqueName: \"kubernetes.io/projected/3b703ac5-d445-4378-8d48-eea0ec2563fc-kube-api-access-tkcpc\") pod \"ingress-canary-qgjbc\" (UID: \"3b703ac5-d445-4378-8d48-eea0ec2563fc\") " pod="openshift-ingress-canary/ingress-canary-qgjbc" Oct 30 00:34:23.356969 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.356940 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lbj\" (UniqueName: \"kubernetes.io/projected/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-kube-api-access-t4lbj\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.357238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.357223 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72b5\" (UniqueName: \"kubernetes.io/projected/16d03985-22a9-42a8-a195-638cb40e9e2c-kube-api-access-h72b5\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:34:23.357311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.357243 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxgk\" (UniqueName: \"kubernetes.io/projected/4667c989-ad40-4b50-9d15-c341751fff71-kube-api-access-7wxgk\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.357311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.357246 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7zp\" (UniqueName: \"kubernetes.io/projected/87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e-kube-api-access-pw7zp\") pod \"migrator-66df5d496b-pg7p7\" (UID: \"87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pg7p7" Oct 30 00:34:23.357382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.357361 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nfmc\" (UniqueName: \"kubernetes.io/projected/e868adcd-9f67-4757-b2e1-9cb31a968d02-kube-api-access-4nfmc\") pod \"network-check-target-6ncmx\" (UID: \"e868adcd-9f67-4757-b2e1-9cb31a968d02\") " pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:23.437111 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437074 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysconfig\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.437264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437129 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-system-cni-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.437264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437156 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.437264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437183 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-kubelet\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.437264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437209 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-sys-fs\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.437264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437215 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysconfig\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.437264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437235 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.437264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437245 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-system-cni-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437263 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437279 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-systemd\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437294 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-kubelet\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437301 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437327 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-sys-fs\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437300 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437346 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437339 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437374 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68188523-c3a4-4a55-90a8-0ba680aad7e2-crio-socket\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437387 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-systemd\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437404 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-conf\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437430 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-registration-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437453 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437462 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68188523-c3a4-4a55-90a8-0ba680aad7e2-crio-socket\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437473 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-bin\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.437470 4517 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437502 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437517 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-registration-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437531 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-dbus\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437542 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-conf\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437556 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-k8s-cni-cncf-io\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437555 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-bin\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.437560 4517 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.437573 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.937550897 +0000 UTC m=+1.849678032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : secret "metrics-daemon-secret" not found Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437595 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437594 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-k8s-cni-cncf-io\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437603 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-dbus\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:23.437607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437622 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-multus\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.437702 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.937683286 +0000 UTC m=+1.849810424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437735 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-hostroot\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437753 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-multus\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437782 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-hostroot\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437817 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-system-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437834 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-device-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437850 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-host\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437865 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-systemd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437881 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-var-lib-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437889 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-system-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437909 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-device-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437897 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437930 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-var-lib-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437937 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9-host\") pod \"node-ca-zbzqk\" (UID: \"1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9\") " pod="openshift-image-registry/node-ca-zbzqk" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437942 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-bin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437957 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-systemd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437960 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-cni-bin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437975 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-node-log\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437973 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-cni-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.437997 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-log-socket\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438021 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-node-log\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438021 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-kubernetes\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438050 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-log-socket\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438061 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d03985-22a9-42a8-a195-638cb40e9e2c-hosts-file\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438072 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-kubernetes\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438086 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-sys\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438095 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d03985-22a9-42a8-a195-638cb40e9e2c-hosts-file\") pod \"node-resolver-62mqm\" (UID: \"16d03985-22a9-42a8-a195-638cb40e9e2c\") " pod="openshift-dns/node-resolver-62mqm" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438107 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-lib-modules\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438122 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-kubelet\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438159 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-sys\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438159 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-var-lib-kubelet\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438178 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438204 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cnibin\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438224 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-netns\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438247 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-cnibin\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438248 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75bd79f7-bef7-44d5-9257-8048cc570d2c-host-slash\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438256 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-lib-modules\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438259 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-sysctl-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438274 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-netns\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438274 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438299 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-systemd-units\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438307 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75bd79f7-bef7-44d5-9257-8048cc570d2c-host-slash\") pod \"iptables-alerter-5whwd\" (UID: \"75bd79f7-bef7-44d5-9257-8048cc570d2c\") " pod="openshift-network-operator/iptables-alerter-5whwd" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438326 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438345 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-systemd-units\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438347 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-netns\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438372 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-run-netns\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438383 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-run\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438417 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-kubelet-config\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.438438 4517 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438445 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-socket-dir-parent\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438448 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d0d2d6e-c723-4393-92e5-65c3e8b4f713-kubelet-config\") pod \"global-pull-secret-syncer-kprr7\" (UID: \"4d0d2d6e-c723-4393-92e5-65c3e8b4f713\") " pod="kube-system/global-pull-secret-syncer-kprr7" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438419 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-run\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438474 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-modprobe-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.438507 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.938493155 +0000 UTC m=+1.850620276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438528 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-var-lib-kubelet\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438544 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cnibin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438551 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-socket-dir-parent\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438562 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-etc-modprobe-d\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438559 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-conf-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438581 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-multus-conf-dir\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438605 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438616 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-var-lib-kubelet\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438614 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-cnibin\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438633 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438633 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-host\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438654 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438669 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0349f41-063d-4636-963b-4b88924b3670-host\") pod \"tuned-md6gg\" (UID: \"c0349f41-063d-4636-963b-4b88924b3670\") " pod="openshift-cluster-node-tuning-operator/tuned-md6gg" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438671 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-multus-certs\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.438654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438688 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-host-run-multus-certs\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438725 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-slash\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438744 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6dd358a-8c7b-4a22-8181-fa6b5aa931a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxlcf\" (UID: \"f6dd358a-8c7b-4a22-8181-fa6b5aa931a9\") " pod="openshift-multus/multus-additional-cni-plugins-zxlcf" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438756 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-ovn\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438780 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-netd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438805 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-etc-kubernetes\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438757 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-slash\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438829 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-socket-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438821 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68188523-c3a4-4a55-90a8-0ba680aad7e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438848 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-host-cni-netd\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438857 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1d6ac87-947e-44dd-9a24-1b1df8755f06-etc-kubernetes\") pod \"multus-f92b8\" (UID: \"d1d6ac87-947e-44dd-9a24-1b1df8755f06\") " pod="openshift-multus/multus-f92b8" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438881 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-run-ovn\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438851 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-etc-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438901 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4667c989-ad40-4b50-9d15-c341751fff71-socket-dir\") pod \"aws-ebs-csi-driver-node-b8qkh\" (UID: \"4667c989-ad40-4b50-9d15-c341751fff71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8qkh" Oct 30 00:34:23.440606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.438937 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b47aade-5b0f-4ab4-ac11-e71acc17a9cb-etc-openvswitch\") pod \"ovnkube-node-qgg9k\" (UID: \"2b47aade-5b0f-4ab4-ac11-e71acc17a9cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:23.445234 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.445187 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:23.520582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.520564 4517 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:34:23.559756 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.559662 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:23.915352 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.915277 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-62mqm_16d03985-22a9-42a8-a195-638cb40e9e2c/dns-node-resolver/0.log" Oct 30 00:34:23.943442 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.943031 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:23.943442 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.943129 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:23.943442 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:23.943189 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:23.944621 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.943530 4517 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:23.944621 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.943615 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.943587039 +0000 UTC m=+2.855714177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:23.944621 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.944404 4517 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:23.944621 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.944465 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.9444492 +0000 UTC m=+2.856576351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:23.944621 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.944571 4517 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:23.944621 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:23.944612 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.944601285 +0000 UTC m=+2.856728417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : secret "metrics-daemon-secret" not found Oct 30 00:34:24.298957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:24.298921 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-fqjft" event={"ID":"041a2fd8-4580-4074-a167-7cb444d727e0","Type":"ContainerStarted","Data":"9567454d74fc6fd193811d34ada13012807d59c5a2fa443588bf8cd3d8e728a5"} Oct 30 00:34:24.821020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:24.820988 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:24.928984 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:24.928937 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:24.950395 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:24.950350 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:24.950588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:24.950429 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:24.950588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:24.950477 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:24.950588 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:24.950480 4517 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:24.950588 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:24.950539 4517 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:24.950588 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:24.950567 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.950549038 +0000 UTC m=+4.862676162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:24.950588 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:24.950585 4517 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:24.950834 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:24.950612 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls podName:68188523-c3a4-4a55-90a8-0ba680aad7e2 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.95059337 +0000 UTC m=+4.862720499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lj5vr" (UID: "68188523-c3a4-4a55-90a8-0ba680aad7e2") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:24.950834 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:24.950634 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.95061951 +0000 UTC m=+4.862746631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : secret "metrics-daemon-secret" not found Oct 30 00:34:25.049135 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:25.049096 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:25.158547 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:25.158463 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:25.268187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:25.268154 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:25.315875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:25.315848 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zbzqk_1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9/node-ca/0.log" Oct 30 00:34:25.690310 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:25.690269 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jz5dq" Oct 30 00:34:25.915777 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:25.915741 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qgjbc_3b703ac5-d445-4378-8d48-eea0ec2563fc/serve-healthcheck-canary/0.log" Oct 30 00:34:26.137851 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:26.137815 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-pg7p7_87c5d9af-1ffa-4bcc-a983-ddea4a8cba3e/migrator/0.log" Oct 30 00:34:26.438794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:26.438686 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:26.646381 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:26.646340 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:26.847858 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:26.847820 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:26.967336 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:26.967305 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:26.967531 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:26.967354 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:26.967531 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:26.967373 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:26.967531 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:26.967476 4517 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:26.967686 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:26.967539 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:30.967525593 +0000 UTC m=+8.879652715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : secret "metrics-daemon-secret" not found Oct 30 00:34:26.967686 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:26.967476 4517 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:26.967686 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:26.967648 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:30.967628077 +0000 UTC m=+8.879755207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:26.967817 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:26.967739 4517 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:34:26.969921 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:26.969904 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68188523-c3a4-4a55-90a8-0ba680aad7e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lj5vr\" (UID: \"68188523-c3a4-4a55-90a8-0ba680aad7e2\") " pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:27.043599 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:27.043566 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:27.118908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:27.118825 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lj5vr" Oct 30 00:34:27.242742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:27.242683 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:27.415662 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:27.415578 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lj5vr"] Oct 30 00:34:27.420860 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:27.420833 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68188523_c3a4_4a55_90a8_0ba680aad7e2.slice/crio-351e7697f1298853329b83416088f8781a1128a3742768e7bc8d72cba3785abc WatchSource:0}: Error finding container 351e7697f1298853329b83416088f8781a1128a3742768e7bc8d72cba3785abc: Status 404 returned error can't find the container with id 351e7697f1298853329b83416088f8781a1128a3742768e7bc8d72cba3785abc Oct 30 00:34:27.899221 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:27.899188 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:28.312616 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:28.312580 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lj5vr" event={"ID":"68188523-c3a4-4a55-90a8-0ba680aad7e2","Type":"ContainerStarted","Data":"8785ebba77931d0b8aa05ef2c560c5f91f650251b879344ca8ed604be6b6ab8d"} Oct 30 00:34:28.312801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:28.312640 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lj5vr" event={"ID":"68188523-c3a4-4a55-90a8-0ba680aad7e2","Type":"ContainerStarted","Data":"bd2dde4ee015d612f0925857f92cba13383ed03aedda72cdd828ef115501c2ef"} Oct 30 00:34:28.312801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:28.312659 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lj5vr" event={"ID":"68188523-c3a4-4a55-90a8-0ba680aad7e2","Type":"ContainerStarted","Data":"351e7697f1298853329b83416088f8781a1128a3742768e7bc8d72cba3785abc"} Oct 30 00:34:28.442446 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:28.442395 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:28.550750 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:28.550697 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:28.662974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:28.662892 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:28.842817 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:28.842784 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:29.046075 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:29.046041 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:29.246454 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:29.246426 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:29.318981 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:29.318892 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lj5vr" event={"ID":"68188523-c3a4-4a55-90a8-0ba680aad7e2","Type":"ContainerStarted","Data":"6141b0376dfd5c5a02823e850e8ee2e87928a52abd5ef822abe4980ff724b6e1"} Oct 30 00:34:29.506524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:29.506487 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:29.613046 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:29.612963 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:29.724675 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:29.724643 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:29.922055 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:29.921973 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:30.447480 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:30.447446 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:30.556688 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:30.556662 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:30.672109 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:30.672079 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:30.757072 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:30.757002 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:34:30.839884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:30.839853 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:30.997427 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:30.997391 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:30.997605 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:30.997431 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:30.997605 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:30.997530 4517 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:30.997605 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:30.997557 4517 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:30.997605 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:30.997585 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs podName:61e5f3a5-6460-4b3e-a423-70d0000dd706 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:38.997572424 +0000 UTC m=+16.909699544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs") pod "network-metrics-daemon-87vfm" (UID: "61e5f3a5-6460-4b3e-a423-70d0000dd706") : secret "metrics-daemon-secret" not found Oct 30 00:34:30.997605 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:34:30.997606 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls podName:b43fb340-5191-4f45-b6d9-9916afc9a2fd nodeName:}" failed. No retries permitted until 2026-10-30 00:34:38.997591594 +0000 UTC m=+16.909718728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls") pod "dns-default-tg442" (UID: "b43fb340-5191-4f45-b6d9-9916afc9a2fd") : secret "dns-default-metrics-tls" not found Oct 30 00:34:31.041603 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:31.041570 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:31.243736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:31.243693 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:31.716856 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:31.716813 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:32.443123 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:32.443095 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:32.640844 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:32.640806 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:32.843130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:32.843101 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:33.039377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:33.039340 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:33.175318 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:33.175232 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:33.175480 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:33.175464 4517 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:34:33.242720 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:33.242686 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.129829 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:34.129797 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.238124 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:34.238088 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.348351 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:34.348319 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.455788 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:34.455695 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.635427 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:34.635397 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.747745 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:34.747645 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.859323 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:34.859287 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.992941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:34.992908 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:35.101740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:35.101692 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:35.241916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:35.241862 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:35.844150 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:35.844097 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:36.044007 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.043971 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:36.390868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.390839 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-xmbzq"] Oct 30 00:34:36.393911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.393891 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:36.399275 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.399249 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs"] Oct 30 00:34:36.402062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.402045 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" Oct 30 00:34:36.410062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.410041 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 00:34:36.410177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.410122 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 00:34:36.412940 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.412925 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xcssj\"" Oct 30 00:34:36.421364 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.421344 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xmbzq"] Oct 30 00:34:36.421746 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.421729 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs"] Oct 30 00:34:36.459507 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.459475 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:36.491640 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.491615 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76467d86df-tb287"] Oct 30 00:34:36.494613 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.494594 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.520401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.520371 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hdpz5\"" Oct 30 00:34:36.520401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.520392 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 00:34:36.520598 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.520468 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9b4g8\"" Oct 30 00:34:36.520598 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.520487 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 00:34:36.520598 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.520487 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 00:34:36.520598 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.520513 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 00:34:36.520598 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.520598 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 00:34:36.521118 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.521100 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76467d86df-tb287"] Oct 30 00:34:36.535330 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.535307 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/50c14243-71e3-4c07-986a-6a25bbffa270-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-b2fbs\" (UID: \"50c14243-71e3-4c07-986a-6a25bbffa270\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" Oct 30 00:34:36.535450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.535355 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c39f3ef-0c7f-49d6-a59f-639ac15dcb00-tmp\") pod \"downloads-6dd789c8b7-xmbzq\" (UID: \"9c39f3ef-0c7f-49d6-a59f-639ac15dcb00\") " pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:36.535450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.535380 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skpkf\" (UniqueName: \"kubernetes.io/projected/9c39f3ef-0c7f-49d6-a59f-639ac15dcb00-kube-api-access-skpkf\") pod \"downloads-6dd789c8b7-xmbzq\" (UID: \"9c39f3ef-0c7f-49d6-a59f-639ac15dcb00\") " pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:36.636326 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636294 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/50c14243-71e3-4c07-986a-6a25bbffa270-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-b2fbs\" (UID: \"50c14243-71e3-4c07-986a-6a25bbffa270\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" Oct 30 00:34:36.636496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636331 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9175d6a-2922-45c2-9b76-d49d7a775e3d-installation-pull-secrets\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.636496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636385 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9175d6a-2922-45c2-9b76-d49d7a775e3d-image-registry-private-configuration\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.636496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636410 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpkf\" (UniqueName: \"kubernetes.io/projected/9c39f3ef-0c7f-49d6-a59f-639ac15dcb00-kube-api-access-skpkf\") pod \"downloads-6dd789c8b7-xmbzq\" (UID: \"9c39f3ef-0c7f-49d6-a59f-639ac15dcb00\") " pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:36.636496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636431 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9175d6a-2922-45c2-9b76-d49d7a775e3d-registry-certificates\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.636674 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636582 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9175d6a-2922-45c2-9b76-d49d7a775e3d-registry-tls\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.636674 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636604 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9175d6a-2922-45c2-9b76-d49d7a775e3d-bound-sa-token\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.636796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636680 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9175d6a-2922-45c2-9b76-d49d7a775e3d-trusted-ca\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.636796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636775 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c39f3ef-0c7f-49d6-a59f-639ac15dcb00-tmp\") pod \"downloads-6dd789c8b7-xmbzq\" (UID: \"9c39f3ef-0c7f-49d6-a59f-639ac15dcb00\") " pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:36.636876 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636803 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jws\" (UniqueName: \"kubernetes.io/projected/e9175d6a-2922-45c2-9b76-d49d7a775e3d-kube-api-access-j2jws\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.636876 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.636839 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9175d6a-2922-45c2-9b76-d49d7a775e3d-ca-trust-extracted\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.637131 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.637114 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c39f3ef-0c7f-49d6-a59f-639ac15dcb00-tmp\") pod \"downloads-6dd789c8b7-xmbzq\" (UID: \"9c39f3ef-0c7f-49d6-a59f-639ac15dcb00\") " pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:36.638993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.638970 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/50c14243-71e3-4c07-986a-6a25bbffa270-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-b2fbs\" (UID: \"50c14243-71e3-4c07-986a-6a25bbffa270\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" Oct 30 00:34:36.655524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.655463 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:36.659275 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.659254 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skpkf\" (UniqueName: \"kubernetes.io/projected/9c39f3ef-0c7f-49d6-a59f-639ac15dcb00-kube-api-access-skpkf\") pod \"downloads-6dd789c8b7-xmbzq\" (UID: \"9c39f3ef-0c7f-49d6-a59f-639ac15dcb00\") " pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:36.705458 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.705426 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:36.712461 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.712441 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" Oct 30 00:34:36.737769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.737738 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9175d6a-2922-45c2-9b76-d49d7a775e3d-image-registry-private-configuration\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.737951 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.737779 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9175d6a-2922-45c2-9b76-d49d7a775e3d-registry-certificates\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.737951 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.737800 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9175d6a-2922-45c2-9b76-d49d7a775e3d-registry-tls\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.737951 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.737932 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9175d6a-2922-45c2-9b76-d49d7a775e3d-bound-sa-token\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.738232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.737967 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9175d6a-2922-45c2-9b76-d49d7a775e3d-trusted-ca\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.738232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.738028 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jws\" (UniqueName: \"kubernetes.io/projected/e9175d6a-2922-45c2-9b76-d49d7a775e3d-kube-api-access-j2jws\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.738232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.738065 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9175d6a-2922-45c2-9b76-d49d7a775e3d-ca-trust-extracted\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.738232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.738105 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9175d6a-2922-45c2-9b76-d49d7a775e3d-installation-pull-secrets\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.738917 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.738891 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9175d6a-2922-45c2-9b76-d49d7a775e3d-registry-certificates\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.739104 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.739074 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9175d6a-2922-45c2-9b76-d49d7a775e3d-ca-trust-extracted\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.739886 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.739861 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9175d6a-2922-45c2-9b76-d49d7a775e3d-trusted-ca\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.740515 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.740495 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9175d6a-2922-45c2-9b76-d49d7a775e3d-registry-tls\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.740754 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.740736 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9175d6a-2922-45c2-9b76-d49d7a775e3d-installation-pull-secrets\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.741379 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.741365 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9175d6a-2922-45c2-9b76-d49d7a775e3d-image-registry-private-configuration\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.762999 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.762971 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9175d6a-2922-45c2-9b76-d49d7a775e3d-bound-sa-token\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.764963 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.764910 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jws\" (UniqueName: \"kubernetes.io/projected/e9175d6a-2922-45c2-9b76-d49d7a775e3d-kube-api-access-j2jws\") pod \"image-registry-76467d86df-tb287\" (UID: \"e9175d6a-2922-45c2-9b76-d49d7a775e3d\") " pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:36.765620 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.765565 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:36.805042 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:36.805012 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:37.018792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.018689 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs"] Oct 30 00:34:37.019494 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.019457 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xmbzq"] Oct 30 00:34:37.024157 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:37.024125 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c14243_71e3_4c07_986a_6a25bbffa270.slice/crio-8c38d0231738e52cd720e471b9f4e2c9d294c740890e21c839ea9087ef46dddd WatchSource:0}: Error finding container 8c38d0231738e52cd720e471b9f4e2c9d294c740890e21c839ea9087ef46dddd: Status 404 returned error can't find the container with id 8c38d0231738e52cd720e471b9f4e2c9d294c740890e21c839ea9087ef46dddd Oct 30 00:34:37.024671 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:37.024652 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c39f3ef_0c7f_49d6_a59f_639ac15dcb00.slice/crio-2f409da7601ceae61147f83c9adee78aa000dffcf714ebf7014fbc0f8f0c415f WatchSource:0}: Error finding container 2f409da7601ceae61147f83c9adee78aa000dffcf714ebf7014fbc0f8f0c415f: Status 404 returned error can't find the container with id 2f409da7601ceae61147f83c9adee78aa000dffcf714ebf7014fbc0f8f0c415f Oct 30 00:34:37.102767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.102737 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76467d86df-tb287"] Oct 30 00:34:37.107372 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:37.107344 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9175d6a_2922_45c2_9b76_d49d7a775e3d.slice/crio-ff37542d7eb027607c0b1604cf6c0c0a550715f2a98239764dca0c8dea14429d WatchSource:0}: Error finding container ff37542d7eb027607c0b1604cf6c0c0a550715f2a98239764dca0c8dea14429d: Status 404 returned error can't find the container with id ff37542d7eb027607c0b1604cf6c0c0a550715f2a98239764dca0c8dea14429d Oct 30 00:34:37.135392 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.135364 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:37.242698 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.242666 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:37.344725 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.344660 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76467d86df-tb287" event={"ID":"e9175d6a-2922-45c2-9b76-d49d7a775e3d","Type":"ContainerStarted","Data":"7b04fec6464a15cc797b9af09ae35a80d1c330f753817cecf5d0621536b780b8"} Oct 30 00:34:37.344911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.344825 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:37.344911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.344847 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76467d86df-tb287" event={"ID":"e9175d6a-2922-45c2-9b76-d49d7a775e3d","Type":"ContainerStarted","Data":"ff37542d7eb027607c0b1604cf6c0c0a550715f2a98239764dca0c8dea14429d"} Oct 30 00:34:37.346128 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.346102 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" event={"ID":"50c14243-71e3-4c07-986a-6a25bbffa270","Type":"ContainerStarted","Data":"8c38d0231738e52cd720e471b9f4e2c9d294c740890e21c839ea9087ef46dddd"} Oct 30 00:34:37.347314 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.347291 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xmbzq" event={"ID":"9c39f3ef-0c7f-49d6-a59f-639ac15dcb00","Type":"ContainerStarted","Data":"2f409da7601ceae61147f83c9adee78aa000dffcf714ebf7014fbc0f8f0c415f"} Oct 30 00:34:37.386735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.386668 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76467d86df-tb287" podStartSLOduration=1.386654104 podStartE2EDuration="1.386654104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:37.386293461 +0000 UTC m=+15.298420627" watchObservedRunningTime="2026-10-30 00:34:37.386654104 +0000 UTC m=+15.298781241" Oct 30 00:34:37.778816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:37.778749 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgg9k" Oct 30 00:34:38.353089 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:38.353049 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" event={"ID":"50c14243-71e3-4c07-986a-6a25bbffa270","Type":"ContainerStarted","Data":"4cc69423f136feff217a6051c04432e69c6ba91a47ef570b759d68bfcb92cfcc"} Oct 30 00:34:38.389446 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:38.389337 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" podStartSLOduration=1.292327435 podStartE2EDuration="2.38932299s" totalImagesPullingTime="1.096995555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:38.388835243 +0000 UTC m=+16.300962397" watchObservedRunningTime="2026-10-30 00:34:38.38932299 +0000 UTC m=+16.301450131" Oct 30 00:34:39.059146 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.059100 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:39.059644 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.059163 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:39.061911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.061883 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43fb340-5191-4f45-b6d9-9916afc9a2fd-metrics-tls\") pod \"dns-default-tg442\" (UID: \"b43fb340-5191-4f45-b6d9-9916afc9a2fd\") " pod="openshift-dns/dns-default-tg442" Oct 30 00:34:39.062052 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.061883 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e5f3a5-6460-4b3e-a423-70d0000dd706-metrics-certs\") pod \"network-metrics-daemon-87vfm\" (UID: \"61e5f3a5-6460-4b3e-a423-70d0000dd706\") " pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:39.114020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.113984 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87vfm" Oct 30 00:34:39.118348 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.118326 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tg442" Oct 30 00:34:39.356463 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.356380 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" Oct 30 00:34:39.361965 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.361940 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b2fbs" Oct 30 00:34:39.450421 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.450390 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87vfm"] Oct 30 00:34:39.454008 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:39.453246 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tg442"] Oct 30 00:34:39.456039 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:39.456009 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e5f3a5_6460_4b3e_a423_70d0000dd706.slice/crio-b30221e213ef00773e619e5dd7f537d0c51a943eb6a2c17d2b44a9c53d862a20 WatchSource:0}: Error finding container b30221e213ef00773e619e5dd7f537d0c51a943eb6a2c17d2b44a9c53d862a20: Status 404 returned error can't find the container with id b30221e213ef00773e619e5dd7f537d0c51a943eb6a2c17d2b44a9c53d862a20 Oct 30 00:34:39.459558 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:39.459530 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43fb340_5191_4f45_b6d9_9916afc9a2fd.slice/crio-63bad1ab81cabb8f420787b886ee7e3ea1d0db9cd583db853adfe412d9b6a4af WatchSource:0}: Error finding container 63bad1ab81cabb8f420787b886ee7e3ea1d0db9cd583db853adfe412d9b6a4af: Status 404 returned error can't find the container with id 63bad1ab81cabb8f420787b886ee7e3ea1d0db9cd583db853adfe412d9b6a4af Oct 30 00:34:40.040693 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:40.040649 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:40.159535 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:40.159499 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:40.287823 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:40.287782 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:40.363929 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:40.363826 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tg442" event={"ID":"b43fb340-5191-4f45-b6d9-9916afc9a2fd","Type":"ContainerStarted","Data":"63bad1ab81cabb8f420787b886ee7e3ea1d0db9cd583db853adfe412d9b6a4af"} Oct 30 00:34:40.366924 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:40.366897 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87vfm" event={"ID":"61e5f3a5-6460-4b3e-a423-70d0000dd706","Type":"ContainerStarted","Data":"b30221e213ef00773e619e5dd7f537d0c51a943eb6a2c17d2b44a9c53d862a20"} Oct 30 00:34:41.372984 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:41.372942 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87vfm" event={"ID":"61e5f3a5-6460-4b3e-a423-70d0000dd706","Type":"ContainerStarted","Data":"3b15515a8ddc54b7de28734d9ed2d0eee2ec4412065d29563323248852a69dc0"} Oct 30 00:34:41.373425 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:41.373009 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87vfm" event={"ID":"61e5f3a5-6460-4b3e-a423-70d0000dd706","Type":"ContainerStarted","Data":"edc963643858c2d8d075047b891fd6d4188686ef20ef2d7306a19c0af619797a"} Oct 30 00:34:41.836874 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:41.836825 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:41.946429 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:41.946357 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:42.377776 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:42.377727 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tg442" event={"ID":"b43fb340-5191-4f45-b6d9-9916afc9a2fd","Type":"ContainerStarted","Data":"5c8d8f1a21182b97392b927d07999c3af85e80bef1b6fa65f141f8f3ddd20434"} Oct 30 00:34:42.378208 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:42.377832 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tg442" event={"ID":"b43fb340-5191-4f45-b6d9-9916afc9a2fd","Type":"ContainerStarted","Data":"599ef2c40e9e4fc73c791ff0012a62d150781cf00127a22c4f4629e9bae39d86"} Oct 30 00:34:42.784997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:42.784963 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:43.381535 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:43.381495 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tg442" Oct 30 00:34:44.473957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.473923 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:44.686023 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.685986 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:44.782875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.782795 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-khckl"] Oct 30 00:34:44.788260 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.788232 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.801630 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.801606 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:34:44.809771 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.809729 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:44.811931 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.811910 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:34:44.812198 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.812165 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:34:44.812269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.812232 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:34:44.819676 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.819654 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:34:44.819798 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.819701 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x9vrg\"" Oct 30 00:34:44.819896 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.819878 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:34:44.906054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.906015 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-textfile\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.906228 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.906071 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b18504a9-4418-4d89-a4f8-70e909d507f5-sys\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.906228 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.906166 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.906228 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.906203 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-tls\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.906379 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.906232 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-wtmp\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.906379 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.906358 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vnpc\" (UniqueName: \"kubernetes.io/projected/b18504a9-4418-4d89-a4f8-70e909d507f5-kube-api-access-9vnpc\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.906440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.906397 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b18504a9-4418-4d89-a4f8-70e909d507f5-metrics-client-ca\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.906440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.906424 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.906503 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.906459 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b18504a9-4418-4d89-a4f8-70e909d507f5-root\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:44.929203 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:44.929170 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:45.007912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.007879 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b18504a9-4418-4d89-a4f8-70e909d507f5-sys\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008099 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.007921 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008099 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.007943 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-tls\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008099 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008088 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b18504a9-4418-4d89-a4f8-70e909d507f5-sys\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008130 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-wtmp\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008184 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vnpc\" (UniqueName: \"kubernetes.io/projected/b18504a9-4418-4d89-a4f8-70e909d507f5-kube-api-access-9vnpc\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008220 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b18504a9-4418-4d89-a4f8-70e909d507f5-metrics-client-ca\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008247 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008270 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-wtmp\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008283 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b18504a9-4418-4d89-a4f8-70e909d507f5-root\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008328 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b18504a9-4418-4d89-a4f8-70e909d507f5-root\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008358 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-textfile\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008674 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008605 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.008674 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.008666 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-textfile\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.009078 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.009051 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b18504a9-4418-4d89-a4f8-70e909d507f5-metrics-client-ca\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.010907 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.010884 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-tls\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.012514 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.012495 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b18504a9-4418-4d89-a4f8-70e909d507f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.030023 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.029992 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vnpc\" (UniqueName: \"kubernetes.io/projected/b18504a9-4418-4d89-a4f8-70e909d507f5-kube-api-access-9vnpc\") pod \"node-exporter-khckl\" (UID: \"b18504a9-4418-4d89-a4f8-70e909d507f5\") " pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.081912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.081871 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:45.107438 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.107399 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-khckl" Oct 30 00:34:45.122024 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:45.121988 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18504a9_4418_4d89_a4f8_70e909d507f5.slice/crio-7bcd68a9e5be47204a97d19025db6d368f34c6388fc089775bb68371cd4da870 WatchSource:0}: Error finding container 7bcd68a9e5be47204a97d19025db6d368f34c6388fc089775bb68371cd4da870: Status 404 returned error can't find the container with id 7bcd68a9e5be47204a97d19025db6d368f34c6388fc089775bb68371cd4da870 Oct 30 00:34:45.191952 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.191910 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:45.323781 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.323739 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:45.390887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.390813 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-khckl" event={"ID":"b18504a9-4418-4d89-a4f8-70e909d507f5","Type":"ContainerStarted","Data":"7bcd68a9e5be47204a97d19025db6d368f34c6388fc089775bb68371cd4da870"} Oct 30 00:34:45.952174 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:45.952135 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:46.265689 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:46.265601 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:46.376475 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:46.376428 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:46.396909 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:46.396872 4517 generic.go:356] "Generic (PLEG): container finished" podID="b18504a9-4418-4d89-a4f8-70e909d507f5" containerID="748fba1dfc81195c40751f865035bd445b7d619c223f9ee88b1c180183549d6f" exitCode=0 Oct 30 00:34:46.397078 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:46.396920 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-khckl" event={"ID":"b18504a9-4418-4d89-a4f8-70e909d507f5","Type":"ContainerDied","Data":"748fba1dfc81195c40751f865035bd445b7d619c223f9ee88b1c180183549d6f"} Oct 30 00:34:46.528830 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:46.528801 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:46.729201 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:46.729159 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:47.403793 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:47.403752 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-khckl" event={"ID":"b18504a9-4418-4d89-a4f8-70e909d507f5","Type":"ContainerStarted","Data":"7cf062096c8dc62a609824b8194afdc16e9c950fd0d2e14ffd302b8b66755ea4"} Oct 30 00:34:47.404278 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:47.403824 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-khckl" event={"ID":"b18504a9-4418-4d89-a4f8-70e909d507f5","Type":"ContainerStarted","Data":"9e0753ca3b4275fd4ea5fdd1bce2b408f8e8ef7b12bad32f709bd8d5411d5787"} Oct 30 00:34:47.464720 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:47.464661 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-khckl" podStartSLOduration=2.406828028 podStartE2EDuration="3.464647365s" totalImagesPullingTime="974.416005ms" totalInitContainerRuntime="83.403332ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:47.454091809 +0000 UTC m=+25.366218981" watchObservedRunningTime="2026-10-30 00:34:47.464647365 +0000 UTC m=+25.376774506" Oct 30 00:34:47.986269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:47.986226 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:48.095894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:48.095855 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:48.212537 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:48.212501 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:49.521999 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:49.521967 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:49.664517 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:49.664480 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:49.775261 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:49.775187 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:51.729650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:51.729605 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:52.640337 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:52.640299 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:52.759989 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:52.759887 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:53.387301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:53.387280 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tg442" Oct 30 00:34:53.446331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:53.446304 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:53.558204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:53.558165 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:54.433792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:54.433740 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xmbzq" event={"ID":"9c39f3ef-0c7f-49d6-a59f-639ac15dcb00","Type":"ContainerStarted","Data":"fae8eb13f721f34399f983de5afad9dcf9f4534404770a4ed8e68a011705264b"} Oct 30 00:34:54.434325 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:54.434056 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:54.460050 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:54.460019 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-xmbzq" Oct 30 00:34:54.492683 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:54.492629 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-xmbzq" podStartSLOduration=2.170546726 podStartE2EDuration="18.492613229s" totalImagesPullingTime="16.322066503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:54.481768787 +0000 UTC m=+32.393895926" watchObservedRunningTime="2026-10-30 00:34:54.492613229 +0000 UTC m=+32.404740401" Oct 30 00:34:54.730186 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:54.730108 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:54.841452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:54.840729 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:55.941060 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:55.941022 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c74974bd5-8629t"] Oct 30 00:34:55.976073 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:55.976042 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c74974bd5-8629t"] Oct 30 00:34:55.976260 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:55.976171 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:55.989357 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:55.989331 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 00:34:55.989534 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:55.989514 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 00:34:55.999930 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:55.999907 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 00:34:56.000078 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:55.999999 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 00:34:56.000078 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.000037 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 00:34:56.000293 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.000276 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dff75\"" Oct 30 00:34:56.022542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.022511 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 00:34:56.100844 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.100809 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c065d29-4cbb-4da6-95df-46aaac175b0e-tmp\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.100844 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.100845 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-oauth-serving-cert\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.101117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.100865 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-service-ca\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.101117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.100895 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-config\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.101117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.100958 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-trusted-ca-bundle\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.101117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.100985 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6gn\" (UniqueName: \"kubernetes.io/projected/5c065d29-4cbb-4da6-95df-46aaac175b0e-kube-api-access-hb6gn\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.101117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.101047 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-oauth-config\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.101117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.101067 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-serving-cert\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.202518 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.202441 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-oauth-config\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.202518 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.202495 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-serving-cert\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.202832 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.202533 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c065d29-4cbb-4da6-95df-46aaac175b0e-tmp\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.202832 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.202561 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-oauth-serving-cert\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.202832 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.202583 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-service-ca\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.202832 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.202616 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-config\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.202832 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.202650 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-trusted-ca-bundle\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.202832 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.202678 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6gn\" (UniqueName: \"kubernetes.io/projected/5c065d29-4cbb-4da6-95df-46aaac175b0e-kube-api-access-hb6gn\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.203537 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.203512 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-oauth-serving-cert\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.203758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.203734 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-config\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.203956 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.203929 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-service-ca\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.204102 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.203967 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-trusted-ca-bundle\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.204206 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.204104 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c065d29-4cbb-4da6-95df-46aaac175b0e-tmp\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.205662 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.205641 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-oauth-config\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.208404 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.208382 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-serving-cert\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.224351 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.224318 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6gn\" (UniqueName: \"kubernetes.io/projected/5c065d29-4cbb-4da6-95df-46aaac175b0e-kube-api-access-hb6gn\") pod \"console-6c74974bd5-8629t\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.308761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.308724 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:34:56.641380 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.641343 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c74974bd5-8629t"] Oct 30 00:34:56.646176 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:34:56.646148 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c065d29_4cbb_4da6_95df_46aaac175b0e.slice/crio-902fd7f4b9ec3dc44e0d17b113b0f5d32f7e86f7f25904acd9d4281bf3a11998 WatchSource:0}: Error finding container 902fd7f4b9ec3dc44e0d17b113b0f5d32f7e86f7f25904acd9d4281bf3a11998: Status 404 returned error can't find the container with id 902fd7f4b9ec3dc44e0d17b113b0f5d32f7e86f7f25904acd9d4281bf3a11998 Oct 30 00:34:56.810555 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.810516 4517 patch_prober.go:29] interesting pod/image-registry-76467d86df-tb287 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 00:34:56.810764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:56.810585 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-76467d86df-tb287" podUID="e9175d6a-2922-45c2-9b76-d49d7a775e3d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:57.323031 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:57.322992 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:57.438737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:57.438539 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:57.447718 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:57.447679 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c74974bd5-8629t" event={"ID":"5c065d29-4cbb-4da6-95df-46aaac175b0e","Type":"ContainerStarted","Data":"902fd7f4b9ec3dc44e0d17b113b0f5d32f7e86f7f25904acd9d4281bf3a11998"} Oct 30 00:34:57.902536 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:57.902499 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6ncmx" Oct 30 00:34:58.359574 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:58.359542 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76467d86df-tb287" Oct 30 00:34:58.546218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:58.546178 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:58.663426 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:58.663306 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:59.932279 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:34:59.932234 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:00.053673 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:00.053613 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:00.462770 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:00.462735 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c74974bd5-8629t" event={"ID":"5c065d29-4cbb-4da6-95df-46aaac175b0e","Type":"ContainerStarted","Data":"013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f"} Oct 30 00:35:00.501740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:00.501571 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c74974bd5-8629t" podStartSLOduration=1.833349225 podStartE2EDuration="5.5015531s" totalImagesPullingTime="3.668203875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:00.501249209 +0000 UTC m=+38.413376352" watchObservedRunningTime="2026-10-30 00:35:00.5015531 +0000 UTC m=+38.413680232" Oct 30 00:35:00.761307 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:35:00.761229 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:35:06.309912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:06.309810 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:35:06.309912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:06.309871 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:35:06.314580 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:06.314554 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:35:06.487614 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:06.487589 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:35:08.444979 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:08.444940 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:08.557113 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:08.557082 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:09.359036 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:09.359010 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:09.469381 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:09.469354 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:10.177689 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:10.177655 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:10.287725 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:10.287691 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:10.533960 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:10.533883 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:10.647364 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:10.647329 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:11.892043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:11.892012 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:12.036913 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:12.036881 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:12.398291 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:12.398261 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:12.512129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:12.512099 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:22.639238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:22.639203 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:22.816117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:22.816086 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:23.445023 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:23.444988 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:23.559353 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:23.559323 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:30.595122 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:30.595090 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:30.703068 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:30.703039 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:30.756656 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:35:30.756624 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:35:32.711436 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.711404 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-65c48c844f-47cwx"] Oct 30 00:35:32.728296 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.728270 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.736187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.736164 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c48c844f-47cwx"] Oct 30 00:35:32.777608 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.777581 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-trusted-ca-bundle\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.777728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.777611 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/256e8329-af55-4138-8f59-f3caf10b881b-tmp\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.777728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.777635 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-console-config\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.777809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.777702 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/256e8329-af55-4138-8f59-f3caf10b881b-console-serving-cert\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.777809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.777759 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-service-ca\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.777809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.777787 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-oauth-serving-cert\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.777897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.777882 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/256e8329-af55-4138-8f59-f3caf10b881b-console-oauth-config\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.777929 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.777910 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfv98\" (UniqueName: \"kubernetes.io/projected/256e8329-af55-4138-8f59-f3caf10b881b-kube-api-access-kfv98\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.786516 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.786491 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xtmdf-8fqw8"] Oct 30 00:35:32.786825 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.786775 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" podUID="72e30b08-acdb-4c39-9689-018566263138" containerName="insights-gathering" containerID="cri-o://82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0" gracePeriod=30 Oct 30 00:35:32.878403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.878382 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-trusted-ca-bundle\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.878513 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.878410 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/256e8329-af55-4138-8f59-f3caf10b881b-tmp\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.878513 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.878432 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-console-config\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.878589 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.878572 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/256e8329-af55-4138-8f59-f3caf10b881b-console-serving-cert\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.878637 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.878608 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-service-ca\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.878686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.878636 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-oauth-serving-cert\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.878767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.878690 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/256e8329-af55-4138-8f59-f3caf10b881b-console-oauth-config\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.878767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.878756 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfv98\" (UniqueName: \"kubernetes.io/projected/256e8329-af55-4138-8f59-f3caf10b881b-kube-api-access-kfv98\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.879153 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.878793 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/256e8329-af55-4138-8f59-f3caf10b881b-tmp\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.879478 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.879454 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-console-config\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.879804 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.879778 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-oauth-serving-cert\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.879898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.879778 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-service-ca\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.880065 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.880041 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256e8329-af55-4138-8f59-f3caf10b881b-trusted-ca-bundle\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.882362 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.882333 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/256e8329-af55-4138-8f59-f3caf10b881b-console-serving-cert\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.884396 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.884370 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/256e8329-af55-4138-8f59-f3caf10b881b-console-oauth-config\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.896244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.896222 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:32.900717 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.900683 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfv98\" (UniqueName: \"kubernetes.io/projected/256e8329-af55-4138-8f59-f3caf10b881b-kube-api-access-kfv98\") pod \"console-65c48c844f-47cwx\" (UID: \"256e8329-af55-4138-8f59-f3caf10b881b\") " pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:32.981431 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:32.981399 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bpntp-hjpmp"] Oct 30 00:35:33.000088 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.000056 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.003810 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.003784 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bpntp-hjpmp"] Oct 30 00:35:33.010598 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.010567 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:33.051874 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.051847 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:33.065935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.065912 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:35:33.080820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.080798 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-service-ca-bundle\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.080917 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.080830 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-trusted-ca-bundle\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.080917 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.080859 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rcnf\" (UniqueName: \"kubernetes.io/projected/f5ae20c4-98e9-4249-abd1-46a9b7df540e-kube-api-access-4rcnf\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.080991 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.080965 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ae20c4-98e9-4249-abd1-46a9b7df540e-archives-path\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.177972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.177946 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:33.181953 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.181923 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path\") pod \"72e30b08-acdb-4c39-9689-018566263138\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " Oct 30 00:35:33.182077 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182004 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle\") pod \"72e30b08-acdb-4c39-9689-018566263138\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " Oct 30 00:35:33.182077 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182065 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g\" (UniqueName: \"kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g\") pod \"72e30b08-acdb-4c39-9689-018566263138\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " Oct 30 00:35:33.182196 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182096 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle\") pod \"72e30b08-acdb-4c39-9689-018566263138\" (UID: \"72e30b08-acdb-4c39-9689-018566263138\") " Oct 30 00:35:33.182255 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182231 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-service-ca-bundle\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.182306 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182264 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-trusted-ca-bundle\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.182306 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182278 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path" pod "72e30b08-acdb-4c39-9689-018566263138" (UID: "72e30b08-acdb-4c39-9689-018566263138"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:33.182306 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182300 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcnf\" (UniqueName: \"kubernetes.io/projected/f5ae20c4-98e9-4249-abd1-46a9b7df540e-kube-api-access-4rcnf\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.182450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182365 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ae20c4-98e9-4249-abd1-46a9b7df540e-archives-path\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.182450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182411 4517 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/72e30b08-acdb-4c39-9689-018566263138-archives-path\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:35:33.182686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182664 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle" pod "72e30b08-acdb-4c39-9689-018566263138" (UID: "72e30b08-acdb-4c39-9689-018566263138"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:35:33.182795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182685 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle" pod "72e30b08-acdb-4c39-9689-018566263138" (UID: "72e30b08-acdb-4c39-9689-018566263138"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:35:33.182795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.182775 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ae20c4-98e9-4249-abd1-46a9b7df540e-archives-path\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.183394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.183369 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-service-ca-bundle\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.183605 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.183582 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-trusted-ca-bundle\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.184656 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.184638 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g" pod "72e30b08-acdb-4c39-9689-018566263138" (UID: "72e30b08-acdb-4c39-9689-018566263138"). InnerVolumeSpecName "kube-api-access-lzh7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:33.203388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.203367 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcnf\" (UniqueName: \"kubernetes.io/projected/f5ae20c4-98e9-4249-abd1-46a9b7df540e-kube-api-access-4rcnf\") pod \"periodic-gathering-bpntp-hjpmp\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.283360 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.283310 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzh7g\" (UniqueName: \"kubernetes.io/projected/72e30b08-acdb-4c39-9689-018566263138-kube-api-access-lzh7g\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:35:33.283360 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.283331 4517 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-service-ca-bundle\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:35:33.283360 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.283342 4517 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72e30b08-acdb-4c39-9689-018566263138-trusted-ca-bundle\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:35:33.289168 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.289148 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:33.326557 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.326538 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:35:33.373723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.373685 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c48c844f-47cwx"] Oct 30 00:35:33.379184 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:35:33.379160 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256e8329_af55_4138_8f59_f3caf10b881b.slice/crio-f63e633c838413f51c27bf7f63380c8dc6b5f553e17e6aafd24cb3385a58d538 WatchSource:0}: Error finding container f63e633c838413f51c27bf7f63380c8dc6b5f553e17e6aafd24cb3385a58d538: Status 404 returned error can't find the container with id f63e633c838413f51c27bf7f63380c8dc6b5f553e17e6aafd24cb3385a58d538 Oct 30 00:35:33.566778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.566749 4517 generic.go:356] "Generic (PLEG): container finished" podID="72e30b08-acdb-4c39-9689-018566263138" containerID="82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0" exitCode=2 Oct 30 00:35:33.566919 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.566799 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" Oct 30 00:35:33.566919 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.566832 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" event={"ID":"72e30b08-acdb-4c39-9689-018566263138","Type":"ContainerDied","Data":"82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0"} Oct 30 00:35:33.566919 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.566886 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xtmdf-8fqw8" event={"ID":"72e30b08-acdb-4c39-9689-018566263138","Type":"ContainerDied","Data":"cc5b6a01276734b88c6053bdf78ed59fe8f43f248d94c240fd0d79225847bcf2"} Oct 30 00:35:33.566919 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.566910 4517 scope.go:122] "RemoveContainer" containerID="82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0" Oct 30 00:35:33.568376 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.568356 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c48c844f-47cwx" event={"ID":"256e8329-af55-4138-8f59-f3caf10b881b","Type":"ContainerStarted","Data":"f1cc7f0298433621f681b989ce83d7f297e851af09e171853828db298e075f48"} Oct 30 00:35:33.568497 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.568390 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c48c844f-47cwx" event={"ID":"256e8329-af55-4138-8f59-f3caf10b881b","Type":"ContainerStarted","Data":"f63e633c838413f51c27bf7f63380c8dc6b5f553e17e6aafd24cb3385a58d538"} Oct 30 00:35:33.575085 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.575002 4517 scope.go:122] "RemoveContainer" containerID="82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0" Oct 30 00:35:33.575319 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:35:33.575300 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0\": container with ID starting with 82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0 not found: ID does not exist" containerID="82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0" Oct 30 00:35:33.575394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.575331 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0"} err="failed to get container status \"82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0\": rpc error: code = NotFound desc = could not find container \"82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0\": container with ID starting with 82d20b33fb3187e8ca5d934940497942f6fa7f7a54bd8d686948fdee18b98be0 not found: ID does not exist" Oct 30 00:35:33.610035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.610012 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xtmdf-8fqw8"] Oct 30 00:35:33.616330 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.616311 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-xtmdf-8fqw8"] Oct 30 00:35:33.648097 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.648074 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bpntp-hjpmp"] Oct 30 00:35:33.652573 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:35:33.652543 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ae20c4_98e9_4249_abd1_46a9b7df540e.slice/crio-fb26f96eede14161a42d44632cbfa78e2acd2b74526b199ea4c82950e0717ed9 WatchSource:0}: Error finding container fb26f96eede14161a42d44632cbfa78e2acd2b74526b199ea4c82950e0717ed9: Status 404 returned error can't find the container with id fb26f96eede14161a42d44632cbfa78e2acd2b74526b199ea4c82950e0717ed9 Oct 30 00:35:33.653095 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:33.653059 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-65c48c844f-47cwx" podStartSLOduration=1.653046111 podStartE2EDuration="1.653046111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:33.652941921 +0000 UTC m=+71.565069085" watchObservedRunningTime="2026-10-30 00:35:33.653046111 +0000 UTC m=+71.565173252" Oct 30 00:35:34.232499 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:34.232464 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e30b08-acdb-4c39-9689-018566263138" path="/var/lib/kubelet/pods/72e30b08-acdb-4c39-9689-018566263138/volumes" Oct 30 00:35:34.573878 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:34.573844 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bpntp-hjpmp" event={"ID":"f5ae20c4-98e9-4249-abd1-46a9b7df540e","Type":"ContainerStarted","Data":"2775f0dde098401e957c2b75fc308aa95d3b7794f2a494cf8172b080bb0bcb9c"} Oct 30 00:35:34.574063 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:34.573899 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bpntp-hjpmp" event={"ID":"f5ae20c4-98e9-4249-abd1-46a9b7df540e","Type":"ContainerStarted","Data":"fb26f96eede14161a42d44632cbfa78e2acd2b74526b199ea4c82950e0717ed9"} Oct 30 00:35:34.609583 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:34.609543 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bpntp-hjpmp" podStartSLOduration=2.609531215 podStartE2EDuration="2.609531215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:34.609434249 +0000 UTC m=+72.521561389" watchObservedRunningTime="2026-10-30 00:35:34.609531215 +0000 UTC m=+72.521658365" Oct 30 00:35:38.440516 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:38.440464 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:38.553382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:38.553353 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:43.052369 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:43.052336 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:43.052773 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:43.052441 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:43.056964 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:43.056944 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:43.610443 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:43.610416 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65c48c844f-47cwx" Oct 30 00:35:43.676922 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:43.676890 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c74974bd5-8629t"] Oct 30 00:35:44.371684 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:44.371655 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:44.511280 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:44.511243 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:53.443368 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:53.443334 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:53.558868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:53.558843 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:57.814424 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:35:57.814386 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:00.757152 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:36:00.757111 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:36:05.363898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:05.363854 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:08.696122 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:08.696083 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c74974bd5-8629t" podUID="5c065d29-4cbb-4da6-95df-46aaac175b0e" containerName="console" containerID="cri-o://013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f" gracePeriod=15 Oct 30 00:36:08.930759 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:08.930735 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:36:09.039467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039445 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c065d29-4cbb-4da6-95df-46aaac175b0e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c065d29-4cbb-4da6-95df-46aaac175b0e-tmp\") pod \"5c065d29-4cbb-4da6-95df-46aaac175b0e\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " Oct 30 00:36:09.039615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039482 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-trusted-ca-bundle\") pod \"5c065d29-4cbb-4da6-95df-46aaac175b0e\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " Oct 30 00:36:09.039615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039508 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-service-ca\") pod \"5c065d29-4cbb-4da6-95df-46aaac175b0e\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " Oct 30 00:36:09.039615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039529 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-oauth-config\") pod \"5c065d29-4cbb-4da6-95df-46aaac175b0e\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " Oct 30 00:36:09.039615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039562 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-config\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-config\") pod \"5c065d29-4cbb-4da6-95df-46aaac175b0e\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " Oct 30 00:36:09.039615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039597 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c065d29-4cbb-4da6-95df-46aaac175b0e-kube-api-access-hb6gn\" (UniqueName: \"kubernetes.io/projected/5c065d29-4cbb-4da6-95df-46aaac175b0e-kube-api-access-hb6gn\") pod \"5c065d29-4cbb-4da6-95df-46aaac175b0e\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " Oct 30 00:36:09.039912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039626 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-serving-cert\") pod \"5c065d29-4cbb-4da6-95df-46aaac175b0e\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " Oct 30 00:36:09.039912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039765 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c065d29-4cbb-4da6-95df-46aaac175b0e-tmp" pod "5c065d29-4cbb-4da6-95df-46aaac175b0e" (UID: "5c065d29-4cbb-4da6-95df-46aaac175b0e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:09.040020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039904 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-oauth-serving-cert\") pod \"5c065d29-4cbb-4da6-95df-46aaac175b0e\" (UID: \"5c065d29-4cbb-4da6-95df-46aaac175b0e\") " Oct 30 00:36:09.040020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.039947 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-trusted-ca-bundle" pod "5c065d29-4cbb-4da6-95df-46aaac175b0e" (UID: "5c065d29-4cbb-4da6-95df-46aaac175b0e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:36:09.040020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.040004 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-config" pod "5c065d29-4cbb-4da6-95df-46aaac175b0e" (UID: "5c065d29-4cbb-4da6-95df-46aaac175b0e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:36:09.040020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.040002 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-service-ca" pod "5c065d29-4cbb-4da6-95df-46aaac175b0e" (UID: "5c065d29-4cbb-4da6-95df-46aaac175b0e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:36:09.040227 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.040139 4517 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c065d29-4cbb-4da6-95df-46aaac175b0e-tmp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:09.040279 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.040262 4517 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-trusted-ca-bundle\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:09.040337 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.040283 4517 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:09.040337 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.040299 4517 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-config\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:09.040337 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.040300 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-oauth-serving-cert" pod "5c065d29-4cbb-4da6-95df-46aaac175b0e" (UID: "5c065d29-4cbb-4da6-95df-46aaac175b0e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:36:09.041997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.041972 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-oauth-config" pod "5c065d29-4cbb-4da6-95df-46aaac175b0e" (UID: "5c065d29-4cbb-4da6-95df-46aaac175b0e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:36:09.042091 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.042029 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-serving-cert" pod "5c065d29-4cbb-4da6-95df-46aaac175b0e" (UID: "5c065d29-4cbb-4da6-95df-46aaac175b0e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:36:09.042091 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.042036 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c065d29-4cbb-4da6-95df-46aaac175b0e-kube-api-access-hb6gn" pod "5c065d29-4cbb-4da6-95df-46aaac175b0e" (UID: "5c065d29-4cbb-4da6-95df-46aaac175b0e"). InnerVolumeSpecName "kube-api-access-hb6gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:36:09.140623 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.140593 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6gn\" (UniqueName: \"kubernetes.io/projected/5c065d29-4cbb-4da6-95df-46aaac175b0e-kube-api-access-hb6gn\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:09.140623 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.140620 4517 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-serving-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:09.140623 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.140630 4517 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c065d29-4cbb-4da6-95df-46aaac175b0e-oauth-serving-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:09.140811 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.140637 4517 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c065d29-4cbb-4da6-95df-46aaac175b0e-console-oauth-config\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:09.685783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.685745 4517 generic.go:356] "Generic (PLEG): container finished" podID="5c065d29-4cbb-4da6-95df-46aaac175b0e" containerID="013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f" exitCode=0 Oct 30 00:36:09.686026 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.685797 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c74974bd5-8629t" Oct 30 00:36:09.686026 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.685833 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c74974bd5-8629t" event={"ID":"5c065d29-4cbb-4da6-95df-46aaac175b0e","Type":"ContainerDied","Data":"013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f"} Oct 30 00:36:09.686026 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.685884 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c74974bd5-8629t" event={"ID":"5c065d29-4cbb-4da6-95df-46aaac175b0e","Type":"ContainerDied","Data":"902fd7f4b9ec3dc44e0d17b113b0f5d32f7e86f7f25904acd9d4281bf3a11998"} Oct 30 00:36:09.686026 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.685903 4517 scope.go:122] "RemoveContainer" containerID="013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f" Oct 30 00:36:09.694126 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.694109 4517 scope.go:122] "RemoveContainer" containerID="013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f" Oct 30 00:36:09.694384 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:36:09.694360 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f\": container with ID starting with 013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f not found: ID does not exist" containerID="013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f" Oct 30 00:36:09.694485 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.694389 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f"} err="failed to get container status \"013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f\": rpc error: code = NotFound desc = could not find container \"013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f\": container with ID starting with 013f516015d3460bfa7181774df8e470ee6aedbba251ca1cce8ab96db972165f not found: ID does not exist" Oct 30 00:36:09.730546 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.730523 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c74974bd5-8629t"] Oct 30 00:36:09.740895 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:09.740878 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c74974bd5-8629t"] Oct 30 00:36:10.235220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:10.235185 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c065d29-4cbb-4da6-95df-46aaac175b0e" path="/var/lib/kubelet/pods/5c065d29-4cbb-4da6-95df-46aaac175b0e/volumes" Oct 30 00:36:12.011322 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:12.011292 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:30.756988 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:36:30.756952 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:36:57.837727 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:57.837678 4517 generic.go:356] "Generic (PLEG): container finished" podID="f5ae20c4-98e9-4249-abd1-46a9b7df540e" containerID="2775f0dde098401e957c2b75fc308aa95d3b7794f2a494cf8172b080bb0bcb9c" exitCode=0 Oct 30 00:36:57.838166 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:57.837753 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bpntp-hjpmp" event={"ID":"f5ae20c4-98e9-4249-abd1-46a9b7df540e","Type":"ContainerDied","Data":"2775f0dde098401e957c2b75fc308aa95d3b7794f2a494cf8172b080bb0bcb9c"} Oct 30 00:36:58.972136 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:58.972114 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:36:59.076645 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.076609 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ae20c4-98e9-4249-abd1-46a9b7df540e-kube-api-access-4rcnf\" (UniqueName: \"kubernetes.io/projected/f5ae20c4-98e9-4249-abd1-46a9b7df540e-kube-api-access-4rcnf\") pod \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " Oct 30 00:36:59.076859 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.076665 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-trusted-ca-bundle\") pod \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " Oct 30 00:36:59.076859 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.076695 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ae20c4-98e9-4249-abd1-46a9b7df540e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ae20c4-98e9-4249-abd1-46a9b7df540e-archives-path\") pod \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " Oct 30 00:36:59.076859 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.076744 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-service-ca-bundle\") pod \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\" (UID: \"f5ae20c4-98e9-4249-abd1-46a9b7df540e\") " Oct 30 00:36:59.077094 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.077069 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-trusted-ca-bundle" pod "f5ae20c4-98e9-4249-abd1-46a9b7df540e" (UID: "f5ae20c4-98e9-4249-abd1-46a9b7df540e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:36:59.077148 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.077095 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-service-ca-bundle" pod "f5ae20c4-98e9-4249-abd1-46a9b7df540e" (UID: "f5ae20c4-98e9-4249-abd1-46a9b7df540e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:36:59.077148 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.077088 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ae20c4-98e9-4249-abd1-46a9b7df540e-archives-path" pod "f5ae20c4-98e9-4249-abd1-46a9b7df540e" (UID: "f5ae20c4-98e9-4249-abd1-46a9b7df540e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:36:59.078863 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.078838 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ae20c4-98e9-4249-abd1-46a9b7df540e-kube-api-access-4rcnf" pod "f5ae20c4-98e9-4249-abd1-46a9b7df540e" (UID: "f5ae20c4-98e9-4249-abd1-46a9b7df540e"). InnerVolumeSpecName "kube-api-access-4rcnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:36:59.178108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.178020 4517 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-trusted-ca-bundle\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:59.178108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.178051 4517 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5ae20c4-98e9-4249-abd1-46a9b7df540e-archives-path\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:59.178108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.178064 4517 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ae20c4-98e9-4249-abd1-46a9b7df540e-service-ca-bundle\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:59.178108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.178076 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rcnf\" (UniqueName: \"kubernetes.io/projected/f5ae20c4-98e9-4249-abd1-46a9b7df540e-kube-api-access-4rcnf\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:36:59.844659 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.844625 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bpntp-hjpmp" Oct 30 00:36:59.844841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.844621 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bpntp-hjpmp" event={"ID":"f5ae20c4-98e9-4249-abd1-46a9b7df540e","Type":"ContainerDied","Data":"fb26f96eede14161a42d44632cbfa78e2acd2b74526b199ea4c82950e0717ed9"} Oct 30 00:36:59.844841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:36:59.844761 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb26f96eede14161a42d44632cbfa78e2acd2b74526b199ea4c82950e0717ed9" Oct 30 00:37:00.418236 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:00.418206 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:00.551180 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:00.551150 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:00.756204 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:37:00.756123 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:37:12.144640 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:12.144591 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:23.081654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:23.081623 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:23.204568 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:23.204540 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:26.587641 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:26.587614 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:26.698114 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:26.698081 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:30.509146 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:30.509099 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:30.756847 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:37:30.756815 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:37:33.026286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:33.026256 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:33.134075 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:33.134041 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:44.888724 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:44.888679 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:45.729400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:45.729358 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:48.296625 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:48.296588 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:48.411693 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:48.411664 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:50.807921 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:50.807887 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:50.980063 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:50.980022 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:51.136606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.136524 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:51.256202 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256169 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r"] Oct 30 00:37:51.256457 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256445 4517 cpu_manager.go:414] "containerMap: removing container" podUID="72e30b08-acdb-4c39-9689-018566263138" containerName="insights-gathering" Oct 30 00:37:51.256507 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256459 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e30b08-acdb-4c39-9689-018566263138" containerName="insights-gathering" Oct 30 00:37:51.256507 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256481 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5c065d29-4cbb-4da6-95df-46aaac175b0e" containerName="console" Oct 30 00:37:51.256507 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256488 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c065d29-4cbb-4da6-95df-46aaac175b0e" containerName="console" Oct 30 00:37:51.256507 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256502 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f5ae20c4-98e9-4249-abd1-46a9b7df540e" containerName="insights-gathering" Oct 30 00:37:51.256638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256508 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ae20c4-98e9-4249-abd1-46a9b7df540e" containerName="insights-gathering" Oct 30 00:37:51.256638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256565 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ae20c4-98e9-4249-abd1-46a9b7df540e" containerName="insights-gathering" Oct 30 00:37:51.256638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256574 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e30b08-acdb-4c39-9689-018566263138" containerName="insights-gathering" Oct 30 00:37:51.256638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.256580 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c065d29-4cbb-4da6-95df-46aaac175b0e" containerName="console" Oct 30 00:37:51.259357 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.259340 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:37:51.284658 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.284633 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 00:37:51.285012 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.284969 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-k9p2p\"" Oct 30 00:37:51.285012 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.284971 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 00:37:51.285912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.285886 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r"] Oct 30 00:37:51.295030 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.295009 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:51.344335 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.344298 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4zd\" (UniqueName: \"kubernetes.io/projected/23dedc80-2107-4d18-be9f-85405fcfa02c-kube-api-access-vq4zd\") pod \"cert-manager-webhook-5dbbbf75cc-wmm7r\" (UID: \"23dedc80-2107-4d18-be9f-85405fcfa02c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:37:51.344488 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.344431 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23dedc80-2107-4d18-be9f-85405fcfa02c-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wmm7r\" (UID: \"23dedc80-2107-4d18-be9f-85405fcfa02c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:37:51.445276 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.445185 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4zd\" (UniqueName: \"kubernetes.io/projected/23dedc80-2107-4d18-be9f-85405fcfa02c-kube-api-access-vq4zd\") pod \"cert-manager-webhook-5dbbbf75cc-wmm7r\" (UID: \"23dedc80-2107-4d18-be9f-85405fcfa02c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:37:51.445276 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.445230 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23dedc80-2107-4d18-be9f-85405fcfa02c-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wmm7r\" (UID: \"23dedc80-2107-4d18-be9f-85405fcfa02c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:37:51.466875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.466835 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23dedc80-2107-4d18-be9f-85405fcfa02c-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wmm7r\" (UID: \"23dedc80-2107-4d18-be9f-85405fcfa02c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:37:51.466989 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.466939 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4zd\" (UniqueName: \"kubernetes.io/projected/23dedc80-2107-4d18-be9f-85405fcfa02c-kube-api-access-vq4zd\") pod \"cert-manager-webhook-5dbbbf75cc-wmm7r\" (UID: \"23dedc80-2107-4d18-be9f-85405fcfa02c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:37:51.585331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.585277 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:37:51.879654 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:51.879616 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r"] Oct 30 00:37:51.884203 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:37:51.884171 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23dedc80_2107_4d18_be9f_85405fcfa02c.slice/crio-ac30ad4783d4f53727839126cb51f33eb015def8a377c990313477c0a1d65e4e WatchSource:0}: Error finding container ac30ad4783d4f53727839126cb51f33eb015def8a377c990313477c0a1d65e4e: Status 404 returned error can't find the container with id ac30ad4783d4f53727839126cb51f33eb015def8a377c990313477c0a1d65e4e Oct 30 00:37:52.017176 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:52.017143 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" event={"ID":"23dedc80-2107-4d18-be9f-85405fcfa02c","Type":"ContainerStarted","Data":"ac30ad4783d4f53727839126cb51f33eb015def8a377c990313477c0a1d65e4e"} Oct 30 00:37:53.059658 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:53.059605 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:53.430048 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:53.429966 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:54.901081 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:54.901047 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:55.010252 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:55.010218 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:56.032745 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:56.032682 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" event={"ID":"23dedc80-2107-4d18-be9f-85405fcfa02c","Type":"ContainerStarted","Data":"41df79d480f22320412bf453998b099029c3534d24033f02ccf8abe6a9fa9076"} Oct 30 00:37:56.033148 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:56.032815 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:37:56.071346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:37:56.071303 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" podStartSLOduration=1.858632766 podStartE2EDuration="5.071289523s" totalImagesPullingTime="3.212656757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:56.07086385 +0000 UTC m=+213.982990980" watchObservedRunningTime="2026-10-30 00:37:56.071289523 +0000 UTC m=+213.983416665" Oct 30 00:38:00.756645 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:38:00.756601 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:38:01.487303 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:01.487269 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:01.733352 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:01.733303 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:01.842688 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:01.842659 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:02.038887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:02.038860 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" Oct 30 00:38:02.602229 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:02.602187 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:02.715684 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:02.715656 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:04.582615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.582536 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-zlscb"] Oct 30 00:38:04.585998 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.585977 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" Oct 30 00:38:04.600677 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.600656 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-t7jhw\"" Oct 30 00:38:04.611215 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.611191 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:38:04.611434 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.611409 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 00:38:04.612576 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.612555 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-zlscb"] Oct 30 00:38:04.744533 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.744486 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn8c4\" (UniqueName: \"kubernetes.io/projected/15c09d7d-c159-4dee-bb5a-60f42e3c9d31-kube-api-access-gn8c4\") pod \"openshift-lws-operator-8465994787-zlscb\" (UID: \"15c09d7d-c159-4dee-bb5a-60f42e3c9d31\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" Oct 30 00:38:04.744738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.744553 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c09d7d-c159-4dee-bb5a-60f42e3c9d31-tmp\") pod \"openshift-lws-operator-8465994787-zlscb\" (UID: \"15c09d7d-c159-4dee-bb5a-60f42e3c9d31\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" Oct 30 00:38:04.845361 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.845269 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn8c4\" (UniqueName: \"kubernetes.io/projected/15c09d7d-c159-4dee-bb5a-60f42e3c9d31-kube-api-access-gn8c4\") pod \"openshift-lws-operator-8465994787-zlscb\" (UID: \"15c09d7d-c159-4dee-bb5a-60f42e3c9d31\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" Oct 30 00:38:04.845361 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.845341 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c09d7d-c159-4dee-bb5a-60f42e3c9d31-tmp\") pod \"openshift-lws-operator-8465994787-zlscb\" (UID: \"15c09d7d-c159-4dee-bb5a-60f42e3c9d31\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" Oct 30 00:38:04.845804 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.845786 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c09d7d-c159-4dee-bb5a-60f42e3c9d31-tmp\") pod \"openshift-lws-operator-8465994787-zlscb\" (UID: \"15c09d7d-c159-4dee-bb5a-60f42e3c9d31\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" Oct 30 00:38:04.870694 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.870666 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn8c4\" (UniqueName: \"kubernetes.io/projected/15c09d7d-c159-4dee-bb5a-60f42e3c9d31-kube-api-access-gn8c4\") pod \"openshift-lws-operator-8465994787-zlscb\" (UID: \"15c09d7d-c159-4dee-bb5a-60f42e3c9d31\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" Oct 30 00:38:04.896499 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:04.896457 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" Oct 30 00:38:05.189759 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:05.189653 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-zlscb"] Oct 30 00:38:05.194407 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:38:05.194372 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15c09d7d_c159_4dee_bb5a_60f42e3c9d31.slice/crio-ca08e491e457201ca28d37f535ad3a1fe90a75656efd05d49855bcdfa20e64b5 WatchSource:0}: Error finding container ca08e491e457201ca28d37f535ad3a1fe90a75656efd05d49855bcdfa20e64b5: Status 404 returned error can't find the container with id ca08e491e457201ca28d37f535ad3a1fe90a75656efd05d49855bcdfa20e64b5 Oct 30 00:38:06.065602 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:06.065565 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" event={"ID":"15c09d7d-c159-4dee-bb5a-60f42e3c9d31","Type":"ContainerStarted","Data":"ca08e491e457201ca28d37f535ad3a1fe90a75656efd05d49855bcdfa20e64b5"} Oct 30 00:38:10.079843 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:10.079807 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" event={"ID":"15c09d7d-c159-4dee-bb5a-60f42e3c9d31","Type":"ContainerStarted","Data":"c5a5b064df7fb55b26026a8f67c4ab1eb8cc4bc5835098893f129bf9510d683d"} Oct 30 00:38:10.116753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:10.116667 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-zlscb" podStartSLOduration=1.83555233 podStartE2EDuration="6.116650891s" totalImagesPullingTime="4.281098561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:10.11610915 +0000 UTC m=+228.028236315" watchObservedRunningTime="2026-10-30 00:38:10.116650891 +0000 UTC m=+228.028778033" Oct 30 00:38:11.189917 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:11.189877 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:11.316926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:11.316894 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:12.259774 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:12.259727 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:13.367138 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:13.367103 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:25.837129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:25.837094 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:25.958382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:25.958347 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5"] Oct 30 00:38:25.965004 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:25.964979 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:25.966187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:25.966163 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:25.984861 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:25.984836 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5"] Oct 30 00:38:26.000224 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.000199 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75b9\" (UniqueName: \"kubernetes.io/projected/771c8810-58df-4215-8198-8fcfce04eb8a-kube-api-access-f75b9\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.000346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.000230 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/771c8810-58df-4215-8198-8fcfce04eb8a-metrics-cert\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.000346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.000257 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/771c8810-58df-4215-8198-8fcfce04eb8a-cert\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.000346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.000275 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/771c8810-58df-4215-8198-8fcfce04eb8a-manager-config\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.059339 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.059319 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 00:38:26.059514 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.059500 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2sxm9\"" Oct 30 00:38:26.059592 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.059526 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 00:38:26.059592 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.059577 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 00:38:26.101027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.100979 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75b9\" (UniqueName: \"kubernetes.io/projected/771c8810-58df-4215-8198-8fcfce04eb8a-kube-api-access-f75b9\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.101027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.101004 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/771c8810-58df-4215-8198-8fcfce04eb8a-metrics-cert\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.101178 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.101033 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/771c8810-58df-4215-8198-8fcfce04eb8a-cert\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.101178 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.101104 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/771c8810-58df-4215-8198-8fcfce04eb8a-manager-config\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.101731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.101688 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/771c8810-58df-4215-8198-8fcfce04eb8a-manager-config\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.103530 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.103509 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/771c8810-58df-4215-8198-8fcfce04eb8a-cert\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.103607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.103546 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/771c8810-58df-4215-8198-8fcfce04eb8a-metrics-cert\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.123035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.123014 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75b9\" (UniqueName: \"kubernetes.io/projected/771c8810-58df-4215-8198-8fcfce04eb8a-kube-api-access-f75b9\") pod \"lws-controller-manager-7fdd95d5df-lbpc5\" (UID: \"771c8810-58df-4215-8198-8fcfce04eb8a\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.277348 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.277323 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:26.577540 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:26.577506 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5"] Oct 30 00:38:26.581519 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:38:26.581496 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771c8810_58df_4215_8198_8fcfce04eb8a.slice/crio-4a71a36225c1a2dd15377a7fba1adcba79f7109d8420a6e248107e13cd7d216f WatchSource:0}: Error finding container 4a71a36225c1a2dd15377a7fba1adcba79f7109d8420a6e248107e13cd7d216f: Status 404 returned error can't find the container with id 4a71a36225c1a2dd15377a7fba1adcba79f7109d8420a6e248107e13cd7d216f Oct 30 00:38:27.136368 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:27.136336 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" event={"ID":"771c8810-58df-4215-8198-8fcfce04eb8a","Type":"ContainerStarted","Data":"4a71a36225c1a2dd15377a7fba1adcba79f7109d8420a6e248107e13cd7d216f"} Oct 30 00:38:27.331328 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:27.331290 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:27.449781 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:27.449692 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:28.134668 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.134625 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:28.301094 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.301066 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr"] Oct 30 00:38:28.305827 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.305804 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.324313 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.324285 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 00:38:28.324512 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.324494 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 00:38:28.324588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.324530 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-725k5\"" Oct 30 00:38:28.324588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.324542 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 00:38:28.324588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.324581 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 00:38:28.324971 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.324951 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 00:38:28.352113 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.352085 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr"] Oct 30 00:38:28.416252 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.416187 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27tts\" (UniqueName: \"kubernetes.io/projected/3476b7bb-1d20-42e9-bdc7-db7d9e919569-kube-api-access-27tts\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.416252 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.416232 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3476b7bb-1d20-42e9-bdc7-db7d9e919569-tmp\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.416438 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.416305 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3476b7bb-1d20-42e9-bdc7-db7d9e919569-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.416438 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.416401 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3476b7bb-1d20-42e9-bdc7-db7d9e919569-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.416543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.416505 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3476b7bb-1d20-42e9-bdc7-db7d9e919569-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.416543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.416533 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3476b7bb-1d20-42e9-bdc7-db7d9e919569-manifests\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.461020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.460991 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:28.517372 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.517346 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3476b7bb-1d20-42e9-bdc7-db7d9e919569-tmp\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.517528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.517376 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3476b7bb-1d20-42e9-bdc7-db7d9e919569-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.517528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.517404 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3476b7bb-1d20-42e9-bdc7-db7d9e919569-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.517528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.517451 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3476b7bb-1d20-42e9-bdc7-db7d9e919569-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.517528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.517467 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3476b7bb-1d20-42e9-bdc7-db7d9e919569-manifests\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.517528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.517500 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27tts\" (UniqueName: \"kubernetes.io/projected/3476b7bb-1d20-42e9-bdc7-db7d9e919569-kube-api-access-27tts\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.517972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.517907 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3476b7bb-1d20-42e9-bdc7-db7d9e919569-manifests\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.520307 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.520282 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3476b7bb-1d20-42e9-bdc7-db7d9e919569-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.520417 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.520354 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3476b7bb-1d20-42e9-bdc7-db7d9e919569-tmp\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.520628 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.520609 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3476b7bb-1d20-42e9-bdc7-db7d9e919569-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.520782 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.520760 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3476b7bb-1d20-42e9-bdc7-db7d9e919569-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.558332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.558309 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27tts\" (UniqueName: \"kubernetes.io/projected/3476b7bb-1d20-42e9-bdc7-db7d9e919569-kube-api-access-27tts\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-cscsr\" (UID: \"3476b7bb-1d20-42e9-bdc7-db7d9e919569\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.654890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.654857 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:28.655419 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.655377 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:28.768265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:28.768198 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:29.061590 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:29.061559 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr"] Oct 30 00:38:29.148796 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:38:29.148767 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3476b7bb_1d20_42e9_bdc7_db7d9e919569.slice/crio-3db266312b2520091b39e5fa1f5af6336255d8fbbde82e6b44066bb73e4e2ed8 WatchSource:0}: Error finding container 3db266312b2520091b39e5fa1f5af6336255d8fbbde82e6b44066bb73e4e2ed8: Status 404 returned error can't find the container with id 3db266312b2520091b39e5fa1f5af6336255d8fbbde82e6b44066bb73e4e2ed8 Oct 30 00:38:30.149535 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:30.149491 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" event={"ID":"771c8810-58df-4215-8198-8fcfce04eb8a","Type":"ContainerStarted","Data":"7baaa919cc1bc2ae904f8a4118cba0eb16346b6824c2466181be28b841fb28db"} Oct 30 00:38:30.150353 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:30.150333 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:30.151895 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:30.151869 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" event={"ID":"3476b7bb-1d20-42e9-bdc7-db7d9e919569","Type":"ContainerStarted","Data":"3db266312b2520091b39e5fa1f5af6336255d8fbbde82e6b44066bb73e4e2ed8"} Oct 30 00:38:30.187413 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:30.187371 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" podStartSLOduration=2.58746177 podStartE2EDuration="5.187360723s" totalImagesPullingTime="2.599898953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:30.186993324 +0000 UTC m=+248.099120467" watchObservedRunningTime="2026-10-30 00:38:30.187360723 +0000 UTC m=+248.099487864" Oct 30 00:38:30.759824 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:38:30.759785 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:38:32.160799 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:32.160755 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" event={"ID":"3476b7bb-1d20-42e9-bdc7-db7d9e919569","Type":"ContainerStarted","Data":"d86ae1edaae75ddf25468494b1a3757214a8c66a0f43c15df88df8572e93696a"} Oct 30 00:38:33.165839 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:33.165802 4517 generic.go:356] "Generic (PLEG): container finished" podID="3476b7bb-1d20-42e9-bdc7-db7d9e919569" containerID="d86ae1edaae75ddf25468494b1a3757214a8c66a0f43c15df88df8572e93696a" exitCode=0 Oct 30 00:38:33.166241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:33.165891 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" event={"ID":"3476b7bb-1d20-42e9-bdc7-db7d9e919569","Type":"ContainerDied","Data":"d86ae1edaae75ddf25468494b1a3757214a8c66a0f43c15df88df8572e93696a"} Oct 30 00:38:34.043846 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:34.043807 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:34.156042 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:34.156009 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:34.170897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:34.170870 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" event={"ID":"3476b7bb-1d20-42e9-bdc7-db7d9e919569","Type":"ContainerStarted","Data":"00ef28df19ac2ce1f1cec5c01dbdce488627b4b77123bfd59255cb05732dfd74"} Oct 30 00:38:34.171260 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:34.171033 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:34.250343 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:34.250299 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" podStartSLOduration=2.64404602 podStartE2EDuration="6.250289782s" totalImagesPullingTime="3.128690898s" totalInitContainerRuntime="477.552864ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:34.239346992 +0000 UTC m=+252.151474142" watchObservedRunningTime="2026-10-30 00:38:34.250289782 +0000 UTC m=+252.162416923" Oct 30 00:38:42.163458 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:42.163427 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-lbpc5" Oct 30 00:38:45.177318 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:45.177286 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-cscsr" Oct 30 00:38:47.614020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:47.613987 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:48.413019 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:48.412963 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:49.559959 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:49.559889 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:49.688503 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:49.688473 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:50.822681 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:50.822650 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:51.051218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.051183 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:51.311266 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.311236 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-t54b4"] Oct 30 00:38:51.314538 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.314517 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:51.339616 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.339594 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 00:38:51.339701 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.339651 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-45tdc\"" Oct 30 00:38:51.340934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.340917 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-t54b4"] Oct 30 00:38:51.372956 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.372930 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:51.386009 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.385989 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/33c577c2-0113-4326-bcf9-bed2991d85f1-webhook-certs\") pod \"odh-observability-584f99c788-t54b4\" (UID: \"33c577c2-0113-4326-bcf9-bed2991d85f1\") " pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:51.386072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.386040 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2l4c\" (UniqueName: \"kubernetes.io/projected/33c577c2-0113-4326-bcf9-bed2991d85f1-kube-api-access-l2l4c\") pod \"odh-observability-584f99c788-t54b4\" (UID: \"33c577c2-0113-4326-bcf9-bed2991d85f1\") " pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:51.486825 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.486786 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2l4c\" (UniqueName: \"kubernetes.io/projected/33c577c2-0113-4326-bcf9-bed2991d85f1-kube-api-access-l2l4c\") pod \"odh-observability-584f99c788-t54b4\" (UID: \"33c577c2-0113-4326-bcf9-bed2991d85f1\") " pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:51.486917 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.486847 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/33c577c2-0113-4326-bcf9-bed2991d85f1-webhook-certs\") pod \"odh-observability-584f99c788-t54b4\" (UID: \"33c577c2-0113-4326-bcf9-bed2991d85f1\") " pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:51.486985 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:38:51.486971 4517 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 00:38:51.487038 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:38:51.487028 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c577c2-0113-4326-bcf9-bed2991d85f1-webhook-certs podName:33c577c2-0113-4326-bcf9-bed2991d85f1 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:51.987011146 +0000 UTC m=+269.899138283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/33c577c2-0113-4326-bcf9-bed2991d85f1-webhook-certs") pod "odh-observability-584f99c788-t54b4" (UID: "33c577c2-0113-4326-bcf9-bed2991d85f1") : secret "odh-observability-webhook-cert" not found Oct 30 00:38:51.510660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.510637 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2l4c\" (UniqueName: \"kubernetes.io/projected/33c577c2-0113-4326-bcf9-bed2991d85f1-kube-api-access-l2l4c\") pod \"odh-observability-584f99c788-t54b4\" (UID: \"33c577c2-0113-4326-bcf9-bed2991d85f1\") " pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:51.918928 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.918898 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:51.991672 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.991646 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/33c577c2-0113-4326-bcf9-bed2991d85f1-webhook-certs\") pod \"odh-observability-584f99c788-t54b4\" (UID: \"33c577c2-0113-4326-bcf9-bed2991d85f1\") " pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:51.994092 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:51.994075 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/33c577c2-0113-4326-bcf9-bed2991d85f1-webhook-certs\") pod \"odh-observability-584f99c788-t54b4\" (UID: \"33c577c2-0113-4326-bcf9-bed2991d85f1\") " pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:52.226819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:52.226758 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:52.520855 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:52.520786 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-t54b4"] Oct 30 00:38:52.525046 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:38:52.525018 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c577c2_0113_4326_bcf9_bed2991d85f1.slice/crio-5aae0ed1271140296a2ff20731e73272fdcd0d895b8eea38b4b66c3ca867562e WatchSource:0}: Error finding container 5aae0ed1271140296a2ff20731e73272fdcd0d895b8eea38b4b66c3ca867562e: Status 404 returned error can't find the container with id 5aae0ed1271140296a2ff20731e73272fdcd0d895b8eea38b4b66c3ca867562e Oct 30 00:38:53.245054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:53.245014 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-t54b4" event={"ID":"33c577c2-0113-4326-bcf9-bed2991d85f1","Type":"ContainerStarted","Data":"5aae0ed1271140296a2ff20731e73272fdcd0d895b8eea38b4b66c3ca867562e"} Oct 30 00:38:53.775054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:53.775022 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:53.932131 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:53.932095 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:54.810024 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:54.809992 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:54.924092 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:54.924066 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:55.193637 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:55.193559 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:55.253786 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:55.253758 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-t54b4" event={"ID":"33c577c2-0113-4326-bcf9-bed2991d85f1","Type":"ContainerStarted","Data":"633e7d03b0c85612daaed10f63b37ded52e66cdff49c487253cf2bc26c7d980e"} Oct 30 00:38:55.253925 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:55.253889 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:38:55.290488 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:55.290444 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-t54b4" podStartSLOduration=2.029220861 podStartE2EDuration="4.290427795s" totalImagesPullingTime="2.261206934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:55.290319184 +0000 UTC m=+273.202446322" watchObservedRunningTime="2026-10-30 00:38:55.290427795 +0000 UTC m=+273.202554936" Oct 30 00:38:55.657268 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:55.657226 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:55.769424 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:38:55.769393 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:00.756790 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:39:00.756754 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:39:03.705972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:03.705928 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:03.906380 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:03.906350 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:06.259769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:06.259734 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-t54b4" Oct 30 00:39:12.386412 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:12.386357 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:19.558942 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:19.558905 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:19.673674 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:19.673639 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:22.507743 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:22.507685 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:30.757270 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:39:30.757223 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:39:47.124190 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:47.124094 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:47.242751 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:47.242718 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:51.593627 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.593596 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:51.709891 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.709847 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:51.833113 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.833075 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:51.858086 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.858020 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-5xnrn"] Oct 30 00:39:51.861549 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.861513 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:39:51.884902 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.884877 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-5xnrn"] Oct 30 00:39:51.901579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.901554 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj"] Oct 30 00:39:51.905033 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.905015 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:39:51.923949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.923923 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtd6\" (UniqueName: \"kubernetes.io/projected/4795d4ba-cbf6-4aa0-b315-f4e283de1229-kube-api-access-gbtd6\") pod \"kserve-controller-manager-776cc66d47-5xnrn\" (UID: \"4795d4ba-cbf6-4aa0-b315-f4e283de1229\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:39:51.924052 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.923959 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkmw2\" (UniqueName: \"kubernetes.io/projected/297b3787-09c8-4c8a-88db-98cc79b4ec9e-kube-api-access-kkmw2\") pod \"llmisvc-controller-manager-869d5dcc5d-gsgxj\" (UID: \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:39:51.924052 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.923981 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4795d4ba-cbf6-4aa0-b315-f4e283de1229-cert\") pod \"kserve-controller-manager-776cc66d47-5xnrn\" (UID: \"4795d4ba-cbf6-4aa0-b315-f4e283de1229\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:39:51.924134 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.924090 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297b3787-09c8-4c8a-88db-98cc79b4ec9e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-gsgxj\" (UID: \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:39:51.933250 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.933219 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj"] Oct 30 00:39:51.948728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.948685 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4rx7v"] Oct 30 00:39:51.951834 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.951817 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:39:51.956997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.956971 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8989t\"" Oct 30 00:39:51.957130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.957001 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 00:39:51.957130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.957058 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 00:39:51.957130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.957122 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2z75g\"" Oct 30 00:39:51.968119 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.968094 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:51.972315 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.972291 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4rx7v"] Oct 30 00:39:51.992470 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.992444 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-52sn7\"" Oct 30 00:39:51.992552 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:51.992531 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 00:39:52.025005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.024982 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtd6\" (UniqueName: \"kubernetes.io/projected/4795d4ba-cbf6-4aa0-b315-f4e283de1229-kube-api-access-gbtd6\") pod \"kserve-controller-manager-776cc66d47-5xnrn\" (UID: \"4795d4ba-cbf6-4aa0-b315-f4e283de1229\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:39:52.025100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.025022 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkmw2\" (UniqueName: \"kubernetes.io/projected/297b3787-09c8-4c8a-88db-98cc79b4ec9e-kube-api-access-kkmw2\") pod \"llmisvc-controller-manager-869d5dcc5d-gsgxj\" (UID: \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:39:52.025100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.025045 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4795d4ba-cbf6-4aa0-b315-f4e283de1229-cert\") pod \"kserve-controller-manager-776cc66d47-5xnrn\" (UID: \"4795d4ba-cbf6-4aa0-b315-f4e283de1229\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:39:52.025179 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.025164 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297b3787-09c8-4c8a-88db-98cc79b4ec9e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-gsgxj\" (UID: \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:39:52.025222 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.025198 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpvv\" (UniqueName: \"kubernetes.io/projected/67e01f0d-253b-4fba-9fea-fe443de7b5ae-kube-api-access-zlpvv\") pod \"model-serving-api-75bd89fdfc-4rx7v\" (UID: \"67e01f0d-253b-4fba-9fea-fe443de7b5ae\") " pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:39:52.025271 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.025227 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e01f0d-253b-4fba-9fea-fe443de7b5ae-tls-certs\") pod \"model-serving-api-75bd89fdfc-4rx7v\" (UID: \"67e01f0d-253b-4fba-9fea-fe443de7b5ae\") " pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:39:52.027611 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.027584 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297b3787-09c8-4c8a-88db-98cc79b4ec9e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-gsgxj\" (UID: \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:39:52.027611 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.027593 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4795d4ba-cbf6-4aa0-b315-f4e283de1229-cert\") pod \"kserve-controller-manager-776cc66d47-5xnrn\" (UID: \"4795d4ba-cbf6-4aa0-b315-f4e283de1229\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:39:52.046288 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.046268 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkmw2\" (UniqueName: \"kubernetes.io/projected/297b3787-09c8-4c8a-88db-98cc79b4ec9e-kube-api-access-kkmw2\") pod \"llmisvc-controller-manager-869d5dcc5d-gsgxj\" (UID: \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:39:52.046381 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.046289 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtd6\" (UniqueName: \"kubernetes.io/projected/4795d4ba-cbf6-4aa0-b315-f4e283de1229-kube-api-access-gbtd6\") pod \"kserve-controller-manager-776cc66d47-5xnrn\" (UID: \"4795d4ba-cbf6-4aa0-b315-f4e283de1229\") " pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:39:52.126353 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.126288 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpvv\" (UniqueName: \"kubernetes.io/projected/67e01f0d-253b-4fba-9fea-fe443de7b5ae-kube-api-access-zlpvv\") pod \"model-serving-api-75bd89fdfc-4rx7v\" (UID: \"67e01f0d-253b-4fba-9fea-fe443de7b5ae\") " pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:39:52.126353 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.126320 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e01f0d-253b-4fba-9fea-fe443de7b5ae-tls-certs\") pod \"model-serving-api-75bd89fdfc-4rx7v\" (UID: \"67e01f0d-253b-4fba-9fea-fe443de7b5ae\") " pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:39:52.128629 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.128607 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e01f0d-253b-4fba-9fea-fe443de7b5ae-tls-certs\") pod \"model-serving-api-75bd89fdfc-4rx7v\" (UID: \"67e01f0d-253b-4fba-9fea-fe443de7b5ae\") " pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:39:52.147234 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.147215 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpvv\" (UniqueName: \"kubernetes.io/projected/67e01f0d-253b-4fba-9fea-fe443de7b5ae-kube-api-access-zlpvv\") pod \"model-serving-api-75bd89fdfc-4rx7v\" (UID: \"67e01f0d-253b-4fba-9fea-fe443de7b5ae\") " pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:39:52.176543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.176526 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:39:52.216672 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.216640 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:39:52.265519 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.265485 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:39:52.392658 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.392589 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:52.473991 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.473963 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-5xnrn"] Oct 30 00:39:52.478048 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:39:52.478014 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4795d4ba_cbf6_4aa0_b315_f4e283de1229.slice/crio-349dee36d28a37066f20c096f28eb6b818eea107a19ed92e85468045f0af8023 WatchSource:0}: Error finding container 349dee36d28a37066f20c096f28eb6b818eea107a19ed92e85468045f0af8023: Status 404 returned error can't find the container with id 349dee36d28a37066f20c096f28eb6b818eea107a19ed92e85468045f0af8023 Oct 30 00:39:52.479346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.479327 4517 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:39:52.522388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.522359 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj"] Oct 30 00:39:52.527675 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:39:52.527652 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod297b3787_09c8_4c8a_88db_98cc79b4ec9e.slice/crio-21d1746d01a8f745f18d5b6a7611feae93ac91074216a11d68b74112133077a7 WatchSource:0}: Error finding container 21d1746d01a8f745f18d5b6a7611feae93ac91074216a11d68b74112133077a7: Status 404 returned error can't find the container with id 21d1746d01a8f745f18d5b6a7611feae93ac91074216a11d68b74112133077a7 Oct 30 00:39:52.571141 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:52.571089 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4rx7v"] Oct 30 00:39:52.575635 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:39:52.575606 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e01f0d_253b_4fba_9fea_fe443de7b5ae.slice/crio-14e2847e10b898cb909ff688c424b7dd549d1b622d65e99c7806b588fdc79d58 WatchSource:0}: Error finding container 14e2847e10b898cb909ff688c424b7dd549d1b622d65e99c7806b588fdc79d58: Status 404 returned error can't find the container with id 14e2847e10b898cb909ff688c424b7dd549d1b622d65e99c7806b588fdc79d58 Oct 30 00:39:53.468266 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:53.468216 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" event={"ID":"297b3787-09c8-4c8a-88db-98cc79b4ec9e","Type":"ContainerStarted","Data":"21d1746d01a8f745f18d5b6a7611feae93ac91074216a11d68b74112133077a7"} Oct 30 00:39:53.472089 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:53.472043 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" event={"ID":"67e01f0d-253b-4fba-9fea-fe443de7b5ae","Type":"ContainerStarted","Data":"14e2847e10b898cb909ff688c424b7dd549d1b622d65e99c7806b588fdc79d58"} Oct 30 00:39:53.474465 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:53.474434 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" event={"ID":"4795d4ba-cbf6-4aa0-b315-f4e283de1229","Type":"ContainerStarted","Data":"349dee36d28a37066f20c096f28eb6b818eea107a19ed92e85468045f0af8023"} Oct 30 00:39:54.373170 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:54.373137 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:54.485629 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:54.485590 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:54.604898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:54.604866 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:54.718987 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:54.718917 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:54.831565 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:54.831536 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:55.488530 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:55.488439 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" event={"ID":"67e01f0d-253b-4fba-9fea-fe443de7b5ae","Type":"ContainerStarted","Data":"1795f6a3047d2f07c5aa2750ba9cc7bd9695b0c3c43d05fd5863303e8ffa1996"} Oct 30 00:39:55.489024 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:55.488588 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:39:55.530397 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:55.530325 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" podStartSLOduration=2.052305568 podStartE2EDuration="4.530311222s" totalImagesPullingTime="2.478005654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:55.52854651 +0000 UTC m=+333.440673649" watchObservedRunningTime="2026-10-30 00:39:55.530311222 +0000 UTC m=+333.442438362" Oct 30 00:39:56.493182 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:56.493147 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" event={"ID":"297b3787-09c8-4c8a-88db-98cc79b4ec9e","Type":"ContainerStarted","Data":"43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536"} Oct 30 00:39:56.493608 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:56.493246 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:39:56.494764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:56.494736 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" event={"ID":"4795d4ba-cbf6-4aa0-b315-f4e283de1229","Type":"ContainerStarted","Data":"fd7b75fda8508d2d4b6fded1de8a024650ada3c84d1f2245acc2cbda61a85c70"} Oct 30 00:39:56.494934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:56.494920 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:39:56.531988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:56.531947 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" podStartSLOduration=2.152027687 podStartE2EDuration="5.531935137s" totalImagesPullingTime="3.37990745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:56.531249157 +0000 UTC m=+334.443376298" watchObservedRunningTime="2026-10-30 00:39:56.531935137 +0000 UTC m=+334.444062278" Oct 30 00:39:56.569130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:39:56.569084 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" podStartSLOduration=2.069689415 podStartE2EDuration="5.569068614s" totalImagesPullingTime="3.499379199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:56.567979608 +0000 UTC m=+334.480106748" watchObservedRunningTime="2026-10-30 00:39:56.569068614 +0000 UTC m=+334.481195755" Oct 30 00:40:00.756579 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:40:00.756535 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:40:06.499890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:06.499858 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-4rx7v" Oct 30 00:40:12.506185 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:12.506145 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:17.893142 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:17.893106 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:27.502499 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:27.502468 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:40:27.505447 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:27.505424 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-5xnrn" Oct 30 00:40:30.757347 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:40:30.757300 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:40:40.986051 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:40.986012 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:41.123669 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:41.123636 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:52.783579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:52.783541 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:55.282912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:55.282877 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:55.744442 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:55.744405 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:56.115242 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:56.115210 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:57.420130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:57.420087 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:59.202694 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:59.202661 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:59.370871 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:59.370839 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:59.673467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:40:59.673435 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:00.758255 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:41:00.758212 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:41:00.812723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:00.812675 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:00.947202 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:00.947168 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:01.772481 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:01.772445 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-wqsz4"] Oct 30 00:41:01.775848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:01.775826 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" Oct 30 00:41:01.796070 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:01.796041 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-wqsz4"] Oct 30 00:41:01.806358 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:01.806310 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 00:41:01.806358 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:01.806356 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 00:41:01.876143 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:01.876109 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-s87k7\"" Oct 30 00:41:01.876311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:01.876280 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4ctw\" (UniqueName: \"kubernetes.io/projected/6dc8574e-42a3-4ad5-855d-83ae930a9846-kube-api-access-h4ctw\") pod \"authorino-operator-68d6dd76dc-wqsz4\" (UID: \"6dc8574e-42a3-4ad5-855d-83ae930a9846\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" Oct 30 00:41:01.977147 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:01.977110 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4ctw\" (UniqueName: \"kubernetes.io/projected/6dc8574e-42a3-4ad5-855d-83ae930a9846-kube-api-access-h4ctw\") pod \"authorino-operator-68d6dd76dc-wqsz4\" (UID: \"6dc8574e-42a3-4ad5-855d-83ae930a9846\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" Oct 30 00:41:01.998941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:01.998909 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4ctw\" (UniqueName: \"kubernetes.io/projected/6dc8574e-42a3-4ad5-855d-83ae930a9846-kube-api-access-h4ctw\") pod \"authorino-operator-68d6dd76dc-wqsz4\" (UID: \"6dc8574e-42a3-4ad5-855d-83ae930a9846\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" Oct 30 00:41:02.087684 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:02.087648 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" Oct 30 00:41:02.339324 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:02.339211 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:02.389877 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:02.389834 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-wqsz4"] Oct 30 00:41:02.394480 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:41:02.394440 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc8574e_42a3_4ad5_855d_83ae930a9846.slice/crio-bd145a6f2cb6c77fa7d1382b3d33916814df2681b30399b39660ea4b680fb8b4 WatchSource:0}: Error finding container bd145a6f2cb6c77fa7d1382b3d33916814df2681b30399b39660ea4b680fb8b4: Status 404 returned error can't find the container with id bd145a6f2cb6c77fa7d1382b3d33916814df2681b30399b39660ea4b680fb8b4 Oct 30 00:41:02.535321 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:02.535281 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:02.742286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:02.742206 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" event={"ID":"6dc8574e-42a3-4ad5-855d-83ae930a9846","Type":"ContainerStarted","Data":"bd145a6f2cb6c77fa7d1382b3d33916814df2681b30399b39660ea4b680fb8b4"} Oct 30 00:41:05.304756 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:05.304719 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:05.421749 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:05.421693 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:05.755615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:05.755579 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" event={"ID":"6dc8574e-42a3-4ad5-855d-83ae930a9846","Type":"ContainerStarted","Data":"6ae0a5c4107350e891952742ffd487a07e0d74640283eb87719b9771f9c4cb95"} Oct 30 00:41:05.755814 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:05.755736 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" Oct 30 00:41:05.793538 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:05.793480 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" podStartSLOduration=1.632100959 podStartE2EDuration="4.793464556s" totalImagesPullingTime="3.161363597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:05.792991323 +0000 UTC m=+403.705118465" watchObservedRunningTime="2026-10-30 00:41:05.793464556 +0000 UTC m=+403.705591696" Oct 30 00:41:06.503025 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:06.502969 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:06.616393 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:06.616352 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:10.711233 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:10.711202 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:10.827450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:10.827413 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:10.938016 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:10.937975 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:11.096241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:11.096204 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:11.209834 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:11.209799 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:12.608423 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:12.608390 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:12.721344 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:12.721312 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:16.526787 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.526753 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr"] Oct 30 00:41:16.530230 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.530208 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:16.544418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.544389 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-87vj2\"" Oct 30 00:41:16.557343 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.557308 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr"] Oct 30 00:41:16.599980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.599941 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/523d16f7-d126-470a-abe4-12b76a960f9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-z9gzr\" (UID: \"523d16f7-d126-470a-abe4-12b76a960f9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:16.599980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.599983 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx54k\" (UniqueName: \"kubernetes.io/projected/523d16f7-d126-470a-abe4-12b76a960f9f-kube-api-access-dx54k\") pod \"kuadrant-operator-controller-manager-956956bc4-z9gzr\" (UID: \"523d16f7-d126-470a-abe4-12b76a960f9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:16.700674 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.700633 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/523d16f7-d126-470a-abe4-12b76a960f9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-z9gzr\" (UID: \"523d16f7-d126-470a-abe4-12b76a960f9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:16.700674 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.700676 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx54k\" (UniqueName: \"kubernetes.io/projected/523d16f7-d126-470a-abe4-12b76a960f9f-kube-api-access-dx54k\") pod \"kuadrant-operator-controller-manager-956956bc4-z9gzr\" (UID: \"523d16f7-d126-470a-abe4-12b76a960f9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:16.701067 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.701046 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/523d16f7-d126-470a-abe4-12b76a960f9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-z9gzr\" (UID: \"523d16f7-d126-470a-abe4-12b76a960f9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:16.727682 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.727646 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx54k\" (UniqueName: \"kubernetes.io/projected/523d16f7-d126-470a-abe4-12b76a960f9f-kube-api-access-dx54k\") pod \"kuadrant-operator-controller-manager-956956bc4-z9gzr\" (UID: \"523d16f7-d126-470a-abe4-12b76a960f9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:16.762408 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.762379 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-wqsz4" Oct 30 00:41:16.842079 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:16.842046 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:17.191960 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:17.191882 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr"] Oct 30 00:41:17.195141 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:41:17.195113 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523d16f7_d126_470a_abe4_12b76a960f9f.slice/crio-55a5b3fad46adc3ef9792a1290fd961e15f88ba0afbaf29816c0dbd25b82544e WatchSource:0}: Error finding container 55a5b3fad46adc3ef9792a1290fd961e15f88ba0afbaf29816c0dbd25b82544e: Status 404 returned error can't find the container with id 55a5b3fad46adc3ef9792a1290fd961e15f88ba0afbaf29816c0dbd25b82544e Oct 30 00:41:17.806542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:17.806506 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" event={"ID":"523d16f7-d126-470a-abe4-12b76a960f9f","Type":"ContainerStarted","Data":"55a5b3fad46adc3ef9792a1290fd961e15f88ba0afbaf29816c0dbd25b82544e"} Oct 30 00:41:18.312810 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:18.312772 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr"] Oct 30 00:41:18.312974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:18.312864 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr"] Oct 30 00:41:18.765923 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:18.765812 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:18.884462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:18.884421 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:21.785346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:21.785312 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:21.825429 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:21.825387 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" podUID="523d16f7-d126-470a-abe4-12b76a960f9f" containerName="manager" containerID="cri-o://057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9" gracePeriod=2 Oct 30 00:41:21.903902 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:21.903869 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:22.077077 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.077048 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:22.088987 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:41:22.088949 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-z9gzr\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="523d16f7-d126-470a-abe4-12b76a960f9f" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:22.149246 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.149207 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523d16f7-d126-470a-abe4-12b76a960f9f-kube-api-access-dx54k\" (UniqueName: \"kubernetes.io/projected/523d16f7-d126-470a-abe4-12b76a960f9f-kube-api-access-dx54k\") pod \"523d16f7-d126-470a-abe4-12b76a960f9f\" (UID: \"523d16f7-d126-470a-abe4-12b76a960f9f\") " Oct 30 00:41:22.149246 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.149250 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523d16f7-d126-470a-abe4-12b76a960f9f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/523d16f7-d126-470a-abe4-12b76a960f9f-extensions-socket-volume\") pod \"523d16f7-d126-470a-abe4-12b76a960f9f\" (UID: \"523d16f7-d126-470a-abe4-12b76a960f9f\") " Oct 30 00:41:22.149571 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.149549 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523d16f7-d126-470a-abe4-12b76a960f9f-extensions-socket-volume" pod "523d16f7-d126-470a-abe4-12b76a960f9f" (UID: "523d16f7-d126-470a-abe4-12b76a960f9f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:22.151628 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.151599 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523d16f7-d126-470a-abe4-12b76a960f9f-kube-api-access-dx54k" pod "523d16f7-d126-470a-abe4-12b76a960f9f" (UID: "523d16f7-d126-470a-abe4-12b76a960f9f"). InnerVolumeSpecName "kube-api-access-dx54k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:22.232903 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.232871 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523d16f7-d126-470a-abe4-12b76a960f9f" path="/var/lib/kubelet/pods/523d16f7-d126-470a-abe4-12b76a960f9f/volumes" Oct 30 00:41:22.240657 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:41:22.240627 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-z9gzr\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="523d16f7-d126-470a-abe4-12b76a960f9f" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:22.249928 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.249898 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx54k\" (UniqueName: \"kubernetes.io/projected/523d16f7-d126-470a-abe4-12b76a960f9f-kube-api-access-dx54k\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:41:22.249928 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.249931 4517 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/523d16f7-d126-470a-abe4-12b76a960f9f-extensions-socket-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:41:22.831155 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.831120 4517 generic.go:356] "Generic (PLEG): container finished" podID="523d16f7-d126-470a-abe4-12b76a960f9f" containerID="057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9" exitCode=2 Oct 30 00:41:22.831665 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.831164 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:22.831665 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.831186 4517 scope.go:122] "RemoveContainer" containerID="057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9" Oct 30 00:41:22.839683 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.839664 4517 scope.go:122] "RemoveContainer" containerID="057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9" Oct 30 00:41:22.840063 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:41:22.840043 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9\": container with ID starting with 057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9 not found: ID does not exist" containerID="057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9" Oct 30 00:41:22.840117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:22.840074 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9"} err="failed to get container status \"057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9\": rpc error: code = NotFound desc = could not find container \"057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9\": container with ID starting with 057143da7539eee2a43cc4f232fad050d09e8ebb015c8817591c30f31cacf6f9 not found: ID does not exist" Oct 30 00:41:22.845377 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:41:22.845356 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-z9gzr\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="523d16f7-d126-470a-abe4-12b76a960f9f" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-z9gzr" Oct 30 00:41:24.356620 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:24.356588 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:24.540141 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:24.540109 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:29.815233 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:29.815197 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:29.939314 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:29.939282 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:30.162162 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:30.162081 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:30.274115 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:30.274085 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:30.563230 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:30.563197 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:30.688065 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:30.688035 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:30.757132 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:41:30.757098 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:41:33.220314 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.220279 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6"] Oct 30 00:41:33.220748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.220591 4517 cpu_manager.go:414] "containerMap: removing container" podUID="523d16f7-d126-470a-abe4-12b76a960f9f" containerName="manager" Oct 30 00:41:33.220748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.220602 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523d16f7-d126-470a-abe4-12b76a960f9f" containerName="manager" Oct 30 00:41:33.220748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.220650 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="523d16f7-d126-470a-abe4-12b76a960f9f" containerName="manager" Oct 30 00:41:33.223527 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.223505 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:33.253579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.253549 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6"] Oct 30 00:41:33.291109 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.291070 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-87vj2\"" Oct 30 00:41:33.348514 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.348474 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gz9r6\" (UID: \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:33.348514 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.348512 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkgw\" (UniqueName: \"kubernetes.io/projected/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-kube-api-access-bvkgw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gz9r6\" (UID: \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:33.449374 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.449337 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gz9r6\" (UID: \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:33.449374 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.449373 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkgw\" (UniqueName: \"kubernetes.io/projected/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-kube-api-access-bvkgw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gz9r6\" (UID: \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:33.449812 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.449792 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gz9r6\" (UID: \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:33.474590 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.474519 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkgw\" (UniqueName: \"kubernetes.io/projected/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-kube-api-access-bvkgw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-gz9r6\" (UID: \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:33.535535 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.535492 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:33.865866 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.865825 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6"] Oct 30 00:41:33.871021 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:41:33.870993 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95695ed_5cf2_4464_b7ca_b3f7481b6a1d.slice/crio-4664146a72eef8a5dbc8268552f1a3e4ca1bd32d35511c51d25d8da00e1b20b2 WatchSource:0}: Error finding container 4664146a72eef8a5dbc8268552f1a3e4ca1bd32d35511c51d25d8da00e1b20b2: Status 404 returned error can't find the container with id 4664146a72eef8a5dbc8268552f1a3e4ca1bd32d35511c51d25d8da00e1b20b2 Oct 30 00:41:33.877364 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:33.877330 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" event={"ID":"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d","Type":"ContainerStarted","Data":"4664146a72eef8a5dbc8268552f1a3e4ca1bd32d35511c51d25d8da00e1b20b2"} Oct 30 00:41:34.134556 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:34.134474 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:34.884075 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:34.884039 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" event={"ID":"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d","Type":"ContainerStarted","Data":"ab0e255649131f75c6bfb64365955fc88dcdacc06256ff2b92a4434635d0cbb9"} Oct 30 00:41:34.884467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:34.884112 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:34.946196 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:34.946142 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" podStartSLOduration=1.946128888 podStartE2EDuration="1.946128888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:34.935143386 +0000 UTC m=+432.847270528" watchObservedRunningTime="2026-10-30 00:41:34.946128888 +0000 UTC m=+432.858256029" Oct 30 00:41:38.633525 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:38.633489 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:38.770256 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:38.770222 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:41.646122 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:41.646084 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:41.756219 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:41.756187 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:45.890331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:45.890301 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 00:41:50.423027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:50.422987 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:50.532907 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:41:50.532867 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:00.757897 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:42:00.757856 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:42:05.263068 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:05.263027 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:05.376278 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:05.376246 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:05.491295 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:05.491259 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:06.187246 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:06.187212 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:06.302749 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:06.302692 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:12.785496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:12.785457 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:25.404538 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:25.404500 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:25.522305 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:25.522266 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:26.493401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:26.493362 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:30.757575 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:42:30.757540 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:42:39.501331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.501250 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:39.626197 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.626164 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:39.795257 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.795223 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bcdgp"] Oct 30 00:42:39.798424 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.798403 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bcdgp" Oct 30 00:42:39.811524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.811503 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8h6tt\"" Oct 30 00:42:39.823304 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.823275 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bcdgp"] Oct 30 00:42:39.869520 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.869485 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbvr\" (UniqueName: \"kubernetes.io/projected/a05a8258-b727-4294-a22d-282c0be469ed-kube-api-access-2dbvr\") pod \"authorino-6f6944bcdd-bcdgp\" (UID: \"a05a8258-b727-4294-a22d-282c0be469ed\") " pod="kuadrant-system/authorino-6f6944bcdd-bcdgp" Oct 30 00:42:39.970149 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.970114 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbvr\" (UniqueName: \"kubernetes.io/projected/a05a8258-b727-4294-a22d-282c0be469ed-kube-api-access-2dbvr\") pod \"authorino-6f6944bcdd-bcdgp\" (UID: \"a05a8258-b727-4294-a22d-282c0be469ed\") " pod="kuadrant-system/authorino-6f6944bcdd-bcdgp" Oct 30 00:42:39.971435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.971409 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bcdgp"] Oct 30 00:42:39.971645 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:42:39.971622 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2dbvr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-bcdgp" podUID="a05a8258-b727-4294-a22d-282c0be469ed" Oct 30 00:42:39.997524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:39.997494 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbvr\" (UniqueName: \"kubernetes.io/projected/a05a8258-b727-4294-a22d-282c0be469ed-kube-api-access-2dbvr\") pod \"authorino-6f6944bcdd-bcdgp\" (UID: \"a05a8258-b727-4294-a22d-282c0be469ed\") " pod="kuadrant-system/authorino-6f6944bcdd-bcdgp" Oct 30 00:42:40.149909 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:40.149834 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bcdgp" Oct 30 00:42:40.157110 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:40.157082 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bcdgp" Oct 30 00:42:40.273172 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:40.273140 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05a8258-b727-4294-a22d-282c0be469ed-kube-api-access-2dbvr\" (UniqueName: \"kubernetes.io/projected/a05a8258-b727-4294-a22d-282c0be469ed-kube-api-access-2dbvr\") pod \"a05a8258-b727-4294-a22d-282c0be469ed\" (UID: \"a05a8258-b727-4294-a22d-282c0be469ed\") " Oct 30 00:42:40.275385 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:40.275364 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05a8258-b727-4294-a22d-282c0be469ed-kube-api-access-2dbvr" pod "a05a8258-b727-4294-a22d-282c0be469ed" (UID: "a05a8258-b727-4294-a22d-282c0be469ed"). InnerVolumeSpecName "kube-api-access-2dbvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:42:40.373904 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:40.373868 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dbvr\" (UniqueName: \"kubernetes.io/projected/a05a8258-b727-4294-a22d-282c0be469ed-kube-api-access-2dbvr\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:42:41.154127 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:41.154098 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bcdgp" Oct 30 00:42:41.218773 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:41.218733 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bcdgp"] Oct 30 00:42:41.225694 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:41.225664 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bcdgp"] Oct 30 00:42:42.233763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:42.233699 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05a8258-b727-4294-a22d-282c0be469ed" path="/var/lib/kubelet/pods/a05a8258-b727-4294-a22d-282c0be469ed/volumes" Oct 30 00:42:46.292963 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:46.292926 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:48.531057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:48.531009 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:48.656911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:48.656875 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:48.769884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:48.769853 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:48.883266 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:48.883230 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:52.185101 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:52.185071 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:52.300409 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:52.300373 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:56.018942 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:56.018885 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:56.151417 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:42:56.151382 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:00.757409 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:43:00.757373 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:43:12.930742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:12.930680 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:24.710413 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:24.710376 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:24.880399 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:24.880365 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:26.077204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:26.077175 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:26.190338 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:26.190305 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:26.301237 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:26.301198 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:30.757551 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:43:30.757512 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:43:36.412875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.412840 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2g96n"] Oct 30 00:43:36.416008 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.415989 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:43:36.439658 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.439629 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2g96n"] Oct 30 00:43:36.507255 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.507220 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a1c725a-d75d-40d7-9f9f-685551855bdc-cert\") pod \"llmisvc-controller-manager-6466c554d8-2g96n\" (UID: \"0a1c725a-d75d-40d7-9f9f-685551855bdc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:43:36.507432 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.507279 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpbh\" (UniqueName: \"kubernetes.io/projected/0a1c725a-d75d-40d7-9f9f-685551855bdc-kube-api-access-4bpbh\") pod \"llmisvc-controller-manager-6466c554d8-2g96n\" (UID: \"0a1c725a-d75d-40d7-9f9f-685551855bdc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:43:36.607614 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.607573 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a1c725a-d75d-40d7-9f9f-685551855bdc-cert\") pod \"llmisvc-controller-manager-6466c554d8-2g96n\" (UID: \"0a1c725a-d75d-40d7-9f9f-685551855bdc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:43:36.607792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.607647 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpbh\" (UniqueName: \"kubernetes.io/projected/0a1c725a-d75d-40d7-9f9f-685551855bdc-kube-api-access-4bpbh\") pod \"llmisvc-controller-manager-6466c554d8-2g96n\" (UID: \"0a1c725a-d75d-40d7-9f9f-685551855bdc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:43:36.610187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.610160 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a1c725a-d75d-40d7-9f9f-685551855bdc-cert\") pod \"llmisvc-controller-manager-6466c554d8-2g96n\" (UID: \"0a1c725a-d75d-40d7-9f9f-685551855bdc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:43:36.629363 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.629332 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpbh\" (UniqueName: \"kubernetes.io/projected/0a1c725a-d75d-40d7-9f9f-685551855bdc-kube-api-access-4bpbh\") pod \"llmisvc-controller-manager-6466c554d8-2g96n\" (UID: \"0a1c725a-d75d-40d7-9f9f-685551855bdc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:43:36.727856 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:36.727787 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:43:37.023798 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:37.023699 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2g96n"] Oct 30 00:43:37.027729 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:43:37.027680 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1c725a_d75d_40d7_9f9f_685551855bdc.slice/crio-b078d23568190ce8ed910b0c0e9de79be44b0188929fba2f857a114fa74f903d WatchSource:0}: Error finding container b078d23568190ce8ed910b0c0e9de79be44b0188929fba2f857a114fa74f903d: Status 404 returned error can't find the container with id b078d23568190ce8ed910b0c0e9de79be44b0188929fba2f857a114fa74f903d Oct 30 00:43:37.368022 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:37.367986 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" event={"ID":"0a1c725a-d75d-40d7-9f9f-685551855bdc","Type":"ContainerStarted","Data":"b078d23568190ce8ed910b0c0e9de79be44b0188929fba2f857a114fa74f903d"} Oct 30 00:43:38.373492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:38.373454 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" event={"ID":"0a1c725a-d75d-40d7-9f9f-685551855bdc","Type":"ContainerStarted","Data":"471015f11a24796c02eae18d80d465d51b85166b9ddbf0b2d676f54a0fc21af5"} Oct 30 00:43:38.373906 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:38.373593 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:43:38.411610 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:43:38.411554 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" podStartSLOduration=2.011521127 podStartE2EDuration="2.411540953s" totalImagesPullingTime="400.019826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:38.411023459 +0000 UTC m=+556.323150602" watchObservedRunningTime="2026-10-30 00:43:38.411540953 +0000 UTC m=+556.323668093" Oct 30 00:44:00.758505 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:44:00.758462 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:44:09.379884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.379803 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2g96n" Oct 30 00:44:09.446892 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.446852 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj"] Oct 30 00:44:09.447166 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.447139 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" podUID="297b3787-09c8-4c8a-88db-98cc79b4ec9e" containerName="manager" containerID="cri-o://43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536" gracePeriod=30 Oct 30 00:44:09.691486 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.691459 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:44:09.771566 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.771529 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/297b3787-09c8-4c8a-88db-98cc79b4ec9e-cert\" (UniqueName: \"kubernetes.io/secret/297b3787-09c8-4c8a-88db-98cc79b4ec9e-cert\") pod \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\" (UID: \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\") " Oct 30 00:44:09.771751 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.771590 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297b3787-09c8-4c8a-88db-98cc79b4ec9e-kube-api-access-kkmw2\" (UniqueName: \"kubernetes.io/projected/297b3787-09c8-4c8a-88db-98cc79b4ec9e-kube-api-access-kkmw2\") pod \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\" (UID: \"297b3787-09c8-4c8a-88db-98cc79b4ec9e\") " Oct 30 00:44:09.773870 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.773842 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297b3787-09c8-4c8a-88db-98cc79b4ec9e-cert" pod "297b3787-09c8-4c8a-88db-98cc79b4ec9e" (UID: "297b3787-09c8-4c8a-88db-98cc79b4ec9e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:09.773870 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.773863 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297b3787-09c8-4c8a-88db-98cc79b4ec9e-kube-api-access-kkmw2" pod "297b3787-09c8-4c8a-88db-98cc79b4ec9e" (UID: "297b3787-09c8-4c8a-88db-98cc79b4ec9e"). InnerVolumeSpecName "kube-api-access-kkmw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:44:09.872537 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.872506 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkmw2\" (UniqueName: \"kubernetes.io/projected/297b3787-09c8-4c8a-88db-98cc79b4ec9e-kube-api-access-kkmw2\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:44:09.872537 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:09.872531 4517 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297b3787-09c8-4c8a-88db-98cc79b4ec9e-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:44:10.305462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.305416 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:10.503121 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.503035 4517 generic.go:356] "Generic (PLEG): container finished" podID="297b3787-09c8-4c8a-88db-98cc79b4ec9e" containerID="43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536" exitCode=0 Oct 30 00:44:10.503121 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.503082 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" Oct 30 00:44:10.503571 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.503124 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" event={"ID":"297b3787-09c8-4c8a-88db-98cc79b4ec9e","Type":"ContainerDied","Data":"43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536"} Oct 30 00:44:10.503571 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.503178 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj" event={"ID":"297b3787-09c8-4c8a-88db-98cc79b4ec9e","Type":"ContainerDied","Data":"21d1746d01a8f745f18d5b6a7611feae93ac91074216a11d68b74112133077a7"} Oct 30 00:44:10.503571 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.503201 4517 scope.go:122] "RemoveContainer" containerID="43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536" Oct 30 00:44:10.511240 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.511221 4517 scope.go:122] "RemoveContainer" containerID="43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536" Oct 30 00:44:10.511515 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:44:10.511496 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536\": container with ID starting with 43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536 not found: ID does not exist" containerID="43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536" Oct 30 00:44:10.511562 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.511524 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536"} err="failed to get container status \"43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536\": rpc error: code = NotFound desc = could not find container \"43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536\": container with ID starting with 43face4a18d257aa48fd1fc9c414219d996b31fdf76d24449a609e17d3671536 not found: ID does not exist" Oct 30 00:44:10.548328 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.548307 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj"] Oct 30 00:44:10.558621 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:10.558601 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gsgxj"] Oct 30 00:44:11.658435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:11.658399 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:12.232897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:12.232864 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297b3787-09c8-4c8a-88db-98cc79b4ec9e" path="/var/lib/kubelet/pods/297b3787-09c8-4c8a-88db-98cc79b4ec9e/volumes" Oct 30 00:44:13.004014 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:13.003983 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:29.957627 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:29.957589 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:30.758402 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:44:30.758365 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:44:32.432448 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:32.432413 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:34.585461 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:34.585413 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:37.321550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:37.321505 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:38.770264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:38.770230 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:43.253273 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:43.253240 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:44.585546 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:44.585509 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:57.284280 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:44:57.284245 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:00.538364 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:00.538313 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:00.729345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:00.729299 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:00.757610 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:45:00.757567 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:45:03.527111 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:03.527072 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:03.670882 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:03.670850 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:05.459048 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:05.459014 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:05.588736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:05.588678 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:06.113302 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.113261 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2"] Oct 30 00:45:06.113751 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.113731 4517 cpu_manager.go:414] "containerMap: removing container" podUID="297b3787-09c8-4c8a-88db-98cc79b4ec9e" containerName="manager" Oct 30 00:45:06.113861 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.113753 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297b3787-09c8-4c8a-88db-98cc79b4ec9e" containerName="manager" Oct 30 00:45:06.113917 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.113862 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="297b3787-09c8-4c8a-88db-98cc79b4ec9e" containerName="manager" Oct 30 00:45:06.117081 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.117057 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.145557 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.145520 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2"] Oct 30 00:45:06.219626 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.219597 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:45:06.219842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.219777 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 00:45:06.219842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.219792 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-76qjj\"" Oct 30 00:45:06.219842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.219795 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:45:06.237724 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.237688 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.237895 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.237742 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.237895 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.237766 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwzmx\" (UniqueName: \"kubernetes.io/projected/f8c57672-5995-4696-ba27-e9d1dd3e46a1-kube-api-access-gwzmx\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.237895 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.237882 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.238031 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.237904 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.238031 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.237938 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.238031 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.237966 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.238031 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.238000 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.238160 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.238055 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339363 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339409 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339429 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339562 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339604 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzmx\" (UniqueName: \"kubernetes.io/projected/f8c57672-5995-4696-ba27-e9d1dd3e46a1-kube-api-access-gwzmx\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339664 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339698 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339817 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339844 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339857 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.339941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.339923 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.340197 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.340042 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.340251 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.340228 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.340505 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.340488 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.342651 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.342624 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.342987 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.342967 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.363067 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.363034 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c57672-5995-4696-ba27-e9d1dd3e46a1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.363375 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.363322 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwzmx\" (UniqueName: \"kubernetes.io/projected/f8c57672-5995-4696-ba27-e9d1dd3e46a1-kube-api-access-gwzmx\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2\" (UID: \"f8c57672-5995-4696-ba27-e9d1dd3e46a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.430090 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.430049 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:06.749760 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.749632 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2"] Oct 30 00:45:06.754229 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:45:06.754203 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c57672_5995_4696_ba27_e9d1dd3e46a1.slice/crio-c9e1aa7bc6a4ece848da82c9c5739d66833e908b26f238e161bc886fd0ec369e WatchSource:0}: Error finding container c9e1aa7bc6a4ece848da82c9c5739d66833e908b26f238e161bc886fd0ec369e: Status 404 returned error can't find the container with id c9e1aa7bc6a4ece848da82c9c5739d66833e908b26f238e161bc886fd0ec369e Oct 30 00:45:06.756061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:06.756043 4517 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:45:07.742570 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:07.742534 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" event={"ID":"f8c57672-5995-4696-ba27-e9d1dd3e46a1","Type":"ContainerStarted","Data":"c9e1aa7bc6a4ece848da82c9c5739d66833e908b26f238e161bc886fd0ec369e"} Oct 30 00:45:10.340244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:10.340213 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:45:10.340504 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:10.340289 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:45:10.340504 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:10.340318 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:45:10.761962 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:10.761868 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" event={"ID":"f8c57672-5995-4696-ba27-e9d1dd3e46a1","Type":"ContainerStarted","Data":"b0a1a2c525a798c53f0c3eb92227605f32cd20b31e4121f3bf5f52df7b840e71"} Oct 30 00:45:10.816733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:10.815130 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" podStartSLOduration=1.231992108 podStartE2EDuration="4.815108497s" totalImagesPullingTime="3.583116389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:10.814068162 +0000 UTC m=+648.726195322" watchObservedRunningTime="2026-10-30 00:45:10.815108497 +0000 UTC m=+648.727235639" Oct 30 00:45:11.431158 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:11.431112 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:11.432772 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:11.432748 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 00:45:11.432897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:11.432797 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" podUID="f8c57672-5995-4696-ba27-e9d1dd3e46a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 00:45:12.430588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:12.430547 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 00:45:12.430797 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:12.430606 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" podUID="f8c57672-5995-4696-ba27-e9d1dd3e46a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 00:45:13.121253 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:13.121219 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:13.234095 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:13.234066 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:13.430766 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:13.430684 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 00:45:13.430766 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:13.430758 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" podUID="f8c57672-5995-4696-ba27-e9d1dd3e46a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 00:45:14.431097 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:14.431058 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 00:45:14.431480 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:14.431117 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" podUID="f8c57672-5995-4696-ba27-e9d1dd3e46a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 00:45:15.430769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:15.430693 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 00:45:15.430968 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:15.430786 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" podUID="f8c57672-5995-4696-ba27-e9d1dd3e46a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 00:45:16.430775 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:16.430730 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 00:45:16.431188 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:16.430795 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" podUID="f8c57672-5995-4696-ba27-e9d1dd3e46a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 00:45:17.430619 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:17.430576 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 00:45:17.430852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:17.430641 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" podUID="f8c57672-5995-4696-ba27-e9d1dd3e46a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 00:45:18.688945 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:18.688910 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:18.689465 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:18.689364 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:18.690263 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:18.690239 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2" Oct 30 00:45:21.195550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.195510 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc"] Oct 30 00:45:21.198989 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.198973 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.228694 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.228665 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc"] Oct 30 00:45:21.276462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.276425 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.276660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.276478 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.276660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.276532 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.276660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.276598 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb26e6-c654-445d-b430-28068567ffc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.276660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.276620 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.276660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.276645 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.276932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.276665 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb26e6-c654-445d-b430-28068567ffc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.276932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.276779 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.276932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.276811 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck444\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-kube-api-access-ck444\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.377680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.377640 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb26e6-c654-445d-b430-28068567ffc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.377680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.377685 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.377935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.377700 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.377935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.377800 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb26e6-c654-445d-b430-28068567ffc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.377935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.377835 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.377935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.377856 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck444\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-kube-api-access-ck444\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.377935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.377887 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.378251 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.377942 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.378251 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.377967 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.378251 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.378189 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.378251 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.378212 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.378543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.378396 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.378543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.378478 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.378842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.378824 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb26e6-c654-445d-b430-28068567ffc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.380177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.380157 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.380302 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.380284 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb26e6-c654-445d-b430-28068567ffc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.401613 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.401577 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.401803 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.401750 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck444\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-kube-api-access-ck444\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsffc\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.512092 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.511999 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:21.848242 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.848211 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc"] Oct 30 00:45:21.851407 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:45:21.851381 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cb26e6_c654_445d_b430_28068567ffc3.slice/crio-47eb1d6a970b7390bbbde32bae7542118c0ac8a5194725ec69311f465d9087c8 WatchSource:0}: Error finding container 47eb1d6a970b7390bbbde32bae7542118c0ac8a5194725ec69311f465d9087c8: Status 404 returned error can't find the container with id 47eb1d6a970b7390bbbde32bae7542118c0ac8a5194725ec69311f465d9087c8 Oct 30 00:45:21.854182 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.854148 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:45:21.854275 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.854214 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:45:21.854275 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:21.854247 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:45:22.835769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:22.835729 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" event={"ID":"c5cb26e6-c654-445d-b430-28068567ffc3","Type":"ContainerStarted","Data":"1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038"} Oct 30 00:45:22.836187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:22.835816 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" event={"ID":"c5cb26e6-c654-445d-b430-28068567ffc3","Type":"ContainerStarted","Data":"47eb1d6a970b7390bbbde32bae7542118c0ac8a5194725ec69311f465d9087c8"} Oct 30 00:45:22.906225 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:22.906165 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" podStartSLOduration=1.906151793 podStartE2EDuration="1.906151793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:22.904839811 +0000 UTC m=+660.816966953" watchObservedRunningTime="2026-10-30 00:45:22.906151793 +0000 UTC m=+660.818278934" Oct 30 00:45:23.513177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:23.513133 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:23.514390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:23.514367 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 00:45:23.514503 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:23.514421 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 00:45:24.513496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:24.513453 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 00:45:24.513908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:24.513514 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 00:45:25.513031 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:25.512988 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 00:45:25.513242 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:25.513048 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 00:45:26.512897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:26.512862 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 00:45:26.513311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:26.512918 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 00:45:27.513056 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:27.513023 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 00:45:27.513446 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:27.513085 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 00:45:28.513063 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:28.513027 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 00:45:28.513464 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:28.513100 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 00:45:30.118622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:30.118586 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:30.119137 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:30.119038 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:30.119807 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:30.119786 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 00:45:30.758195 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:45:30.758158 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:45:30.849922 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:30.849881 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:31.386890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:31.386854 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:33.035788 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:33.035683 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:33.304317 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:33.304235 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:34.555515 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:34.555483 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:34.692198 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:45:34.692158 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:00.757239 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:46:00.757194 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:46:02.010578 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:02.010547 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:02.136909 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:02.136877 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:02.576212 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:02.576169 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:02.701639 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:02.701594 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:02.852587 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:02.852498 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:02.989615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:02.989580 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:03.138505 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.138414 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:03.266968 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.266930 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:03.403877 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.403796 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh"] Oct 30 00:46:03.412250 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.412220 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.418770 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.418730 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:03.431307 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.431266 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh"] Oct 30 00:46:03.439324 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.439298 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2b4105-openshift-default-dockercfg-wzpvv\"" Oct 30 00:46:03.536663 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.536632 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:03.537845 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.537802 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.537920 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.537897 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.537972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.537922 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.537972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.537945 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.538048 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.537999 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.538125 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.538109 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbvws\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-kube-api-access-zbvws\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.538165 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.538140 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.538269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.538250 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.538320 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.538303 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.639921 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.639980 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.640012 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.640045 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.640080 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.640124 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvws\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-kube-api-access-zbvws\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.640163 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.640223 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.640260 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.640733 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.641003 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.641137 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.641631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.641378 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.644857 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.643025 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.657783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.657636 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.658203 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.658021 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.663917 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.663890 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvws\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-kube-api-access-zbvws\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.664209 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.664191 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:03.698564 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.698521 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:03.730593 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:03.730557 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:04.050450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:04.050419 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh"] Oct 30 00:46:04.054871 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:46:04.054840 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5e313a_1534_49cc_bb1c_9ef6a29649e1.slice/crio-032668312c15727e7de2b4181cd0bbd78dae50e280735db381eaaf8e297e16bc WatchSource:0}: Error finding container 032668312c15727e7de2b4181cd0bbd78dae50e280735db381eaaf8e297e16bc: Status 404 returned error can't find the container with id 032668312c15727e7de2b4181cd0bbd78dae50e280735db381eaaf8e297e16bc Oct 30 00:46:04.056993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:04.056962 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:46:04.057082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:04.057023 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:46:04.057082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:04.057053 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:46:05.015749 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:05.015691 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" event={"ID":"8f5e313a-1534-49cc-bb1c-9ef6a29649e1","Type":"ContainerStarted","Data":"3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4"} Oct 30 00:46:05.016280 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:05.015779 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" event={"ID":"8f5e313a-1534-49cc-bb1c-9ef6a29649e1","Type":"ContainerStarted","Data":"032668312c15727e7de2b4181cd0bbd78dae50e280735db381eaaf8e297e16bc"} Oct 30 00:46:05.068327 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:05.068263 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" podStartSLOduration=2.068249804 podStartE2EDuration="2.068249804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:05.068135842 +0000 UTC m=+702.980262985" watchObservedRunningTime="2026-10-30 00:46:05.068249804 +0000 UTC m=+702.980376942" Oct 30 00:46:05.730921 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:05.730883 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:05.736902 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:05.736874 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:06.020629 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:06.020548 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:06.021745 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:06.021697 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:06.187208 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:06.187176 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:06.307433 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:06.307400 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:06.975389 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:06.975355 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:07.100536 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:07.100502 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:07.327211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:07.327174 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:07.486026 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:07.485989 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:07.625778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:07.625653 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:07.762881 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:07.762848 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:07.951035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:07.950932 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:08.112735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:08.112690 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:08.711912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:08.711873 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:08.852895 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:08.852866 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:10.433169 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:10.433124 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:10.554423 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:10.554389 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:11.305734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:11.305671 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:11.439357 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:11.439315 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:11.571259 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:11.571179 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:11.692344 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:11.692288 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:13.266962 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:13.266927 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:13.643050 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:13.643008 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:13.773469 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:13.773440 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:14.514077 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:14.514045 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:14.638149 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:14.638118 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:15.491040 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:15.491003 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:15.607997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:15.607953 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:16.147179 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:16.147130 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:16.287903 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:16.287868 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:17.011005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:17.010952 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:17.135744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:17.135697 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:17.505604 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:17.505572 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:19.149437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:19.149395 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:19.262964 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:19.262931 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:19.387232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:19.387182 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:20.930028 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:20.929994 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:21.042775 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:21.042738 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:21.153315 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:21.153277 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:22.648426 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:22.648392 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:24.110811 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.110775 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd"] Oct 30 00:46:24.114391 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.114367 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.131266 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.131237 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:46:24.136032 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.136004 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd"] Oct 30 00:46:24.139580 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.139553 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-89a676-serving-cert\"" Oct 30 00:46:24.144984 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.144962 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 00:46:24.145546 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.145522 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k5bc5\"" Oct 30 00:46:24.203216 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.203166 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12168cf5-4008-4e5e-9799-fc53b267f3be-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.203414 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.203223 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-maas-api-tls\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.203414 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.203320 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wcdx\" (UniqueName: \"kubernetes.io/projected/12168cf5-4008-4e5e-9799-fc53b267f3be-kube-api-access-8wcdx\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.203585 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.203447 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-metrics-certs\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.211510 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.211480 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:24.234920 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.234894 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:46:24.303995 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.303959 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-maas-api-tls\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.303995 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.303993 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wcdx\" (UniqueName: \"kubernetes.io/projected/12168cf5-4008-4e5e-9799-fc53b267f3be-kube-api-access-8wcdx\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.304257 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.304033 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-metrics-certs\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.304257 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.304097 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12168cf5-4008-4e5e-9799-fc53b267f3be-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.305066 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.304993 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12168cf5-4008-4e5e-9799-fc53b267f3be-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.306906 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.306884 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-metrics-certs\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.307027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.306884 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-maas-api-tls\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.325522 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.325497 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wcdx\" (UniqueName: \"kubernetes.io/projected/12168cf5-4008-4e5e-9799-fc53b267f3be-kube-api-access-8wcdx\") pod \"maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.333657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.333629 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:24.426406 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.426319 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:24.454077 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.454039 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:24.741784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:24.741691 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd"] Oct 30 00:46:24.745581 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:46:24.745557 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12168cf5_4008_4e5e_9799_fc53b267f3be.slice/crio-e73be9e0fd5483c81fdc3404b87e6692d48e7b0dcc1e0fe0d4f3b5a0c452a413 WatchSource:0}: Error finding container e73be9e0fd5483c81fdc3404b87e6692d48e7b0dcc1e0fe0d4f3b5a0c452a413: Status 404 returned error can't find the container with id e73be9e0fd5483c81fdc3404b87e6692d48e7b0dcc1e0fe0d4f3b5a0c452a413 Oct 30 00:46:25.110957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:25.110923 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" event={"ID":"12168cf5-4008-4e5e-9799-fc53b267f3be","Type":"ContainerStarted","Data":"e73be9e0fd5483c81fdc3404b87e6692d48e7b0dcc1e0fe0d4f3b5a0c452a413"} Oct 30 00:46:25.965822 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:25.965784 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:26.259245 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:26.259155 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:26.372034 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:26.372004 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:26.491672 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:26.491633 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:26.702829 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:26.702780 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:26.822564 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:26.822528 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:27.423460 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:27.423326 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:27.545168 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:27.545134 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:28.126546 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:28.126503 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" event={"ID":"12168cf5-4008-4e5e-9799-fc53b267f3be","Type":"ContainerStarted","Data":"9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc"} Oct 30 00:46:28.126763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:28.126649 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:28.164968 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:28.164908 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" podStartSLOduration=1.8081648449999999 podStartE2EDuration="4.164889855s" totalImagesPullingTime="2.35672501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:28.164234055 +0000 UTC m=+726.076361197" watchObservedRunningTime="2026-10-30 00:46:28.164889855 +0000 UTC m=+726.077016998" Oct 30 00:46:28.309842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:28.309813 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:29.399408 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.399365 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4"] Oct 30 00:46:29.403332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.403306 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.427000 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.426952 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"openshift-service-ca.crt\"" Oct 30 00:46:29.427288 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.426996 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"default-dockercfg-5kqq2\"" Oct 30 00:46:29.427288 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.427098 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"simulator-w2-40ac41-kserve-self-signed-certs\"" Oct 30 00:46:29.427421 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.427218 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"kube-root-ca.crt\"" Oct 30 00:46:29.429590 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.429564 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4"] Oct 30 00:46:29.452463 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.452427 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:29.455433 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.455410 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tmp-dir\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.455550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.455457 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4tq\" (UniqueName: \"kubernetes.io/projected/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-kube-api-access-xv4tq\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.455550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.455484 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-model-cache\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.455643 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.455616 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-home\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.455767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.455751 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-dshm\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.455820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.455786 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tls-certs\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.556863 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.556831 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-dshm\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.557047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.556873 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tls-certs\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.557047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.556924 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tmp-dir\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.557047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.556963 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4tq\" (UniqueName: \"kubernetes.io/projected/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-kube-api-access-xv4tq\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.557047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.556988 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-model-cache\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.557047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.557014 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-home\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.557391 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.557366 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tmp-dir\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.557474 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.557423 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-model-cache\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.557521 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.557466 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-home\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.559890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.559867 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-dshm\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.560254 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.560231 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tls-certs\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.578806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.578769 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4tq\" (UniqueName: \"kubernetes.io/projected/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-kube-api-access-xv4tq\") pod \"simulator-w2-40ac41-kserve-fb5b597d4-z58z4\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:29.719761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:29.719653 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:30.017664 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:30.017575 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4"] Oct 30 00:46:30.021648 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:46:30.021621 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod340bbdbe_47e6_43d3_b384_e2f5a8e4b726.slice/crio-933b877a632f5fbb3c9585664917e385befd1b271705d01626b87226d70026a8 WatchSource:0}: Error finding container 933b877a632f5fbb3c9585664917e385befd1b271705d01626b87226d70026a8: Status 404 returned error can't find the container with id 933b877a632f5fbb3c9585664917e385befd1b271705d01626b87226d70026a8 Oct 30 00:46:30.136209 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:30.136177 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" event={"ID":"340bbdbe-47e6-43d3-b384-e2f5a8e4b726","Type":"ContainerStarted","Data":"933b877a632f5fbb3c9585664917e385befd1b271705d01626b87226d70026a8"} Oct 30 00:46:30.233893 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:30.233854 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:30.775300 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:46:30.775259 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:46:32.147439 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:32.147398 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" event={"ID":"340bbdbe-47e6-43d3-b384-e2f5a8e4b726","Type":"ContainerStarted","Data":"3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34"} Oct 30 00:46:32.177069 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:32.177037 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:32.187051 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:32.187003 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podStartSLOduration=1.886321171 podStartE2EDuration="3.18698867s" totalImagesPullingTime="1.300667499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:32.185944767 +0000 UTC m=+730.098071911" watchObservedRunningTime="2026-10-30 00:46:32.18698867 +0000 UTC m=+730.099115804" Oct 30 00:46:32.292862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:32.292819 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:32.720358 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:32.720321 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:32.791898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:32.791857 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:32.915750 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:32.915687 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:33.040935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:33.040904 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:34.136108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:34.136081 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 00:46:34.226589 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:34.226539 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:36.002002 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:36.001970 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:37.450073 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:37.450035 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:37.585829 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:37.585793 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:37.716991 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:37.716908 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:38.052062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:38.052016 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:38.184018 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:38.183978 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:38.293725 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:38.293667 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:39.720277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:39.720233 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:39.735478 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:39.735447 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:39.774398 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:39.774354 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:40.224322 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:40.224294 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:46:41.216696 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.216608 4517 generic.go:356] "Generic (PLEG): container finished" podID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerID="3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4" exitCode=0 Oct 30 00:46:41.216696 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.216682 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" event={"ID":"8f5e313a-1534-49cc-bb1c-9ef6a29649e1","Type":"ContainerDied","Data":"3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4"} Oct 30 00:46:41.217310 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.217110 4517 scope.go:122] "RemoveContainer" containerID="3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4" Oct 30 00:46:41.218344 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.218315 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:46:41.218405 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.218392 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:46:41.218444 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.218432 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:46:41.266939 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.266909 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb"] Oct 30 00:46:41.271588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.271567 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.285358 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.285035 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"default-dockercfg-n8wx7\"" Oct 30 00:46:41.285358 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.285157 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"simulator-w4-6add35-kserve-self-signed-certs\"" Oct 30 00:46:41.285358 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.285248 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"kube-root-ca.crt\"" Oct 30 00:46:41.285358 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.285318 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"openshift-service-ca.crt\"" Oct 30 00:46:41.300694 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.300665 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb"] Oct 30 00:46:41.357640 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.357567 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:41.374588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.374535 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-home\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.375096 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.375072 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-model-cache\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.375357 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.375292 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4dx\" (UniqueName: \"kubernetes.io/projected/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-kube-api-access-2r4dx\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.375497 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.375482 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-dshm\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.375692 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.375646 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tmp-dir\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.375916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.375900 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tls-certs\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.477033 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.476924 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tls-certs\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.477033 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.477015 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-home\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.477296 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.477097 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-model-cache\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.478005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.477459 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-home\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.478005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.477533 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-model-cache\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.478005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.477577 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4dx\" (UniqueName: \"kubernetes.io/projected/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-kube-api-access-2r4dx\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.478005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.477611 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-dshm\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.478005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.477638 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tmp-dir\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.478005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.477996 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tmp-dir\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.479978 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.479952 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-dshm\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.480333 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.480313 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tls-certs\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.495657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.495623 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:41.498853 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.498818 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4dx\" (UniqueName: \"kubernetes.io/projected/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-kube-api-access-2r4dx\") pod \"simulator-w4-6add35-kserve-b78c678d-7bffb\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.590466 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.590433 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:41.891934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:41.891876 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb"] Oct 30 00:46:42.223259 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.223155 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" event={"ID":"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e","Type":"ContainerStarted","Data":"5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0"} Oct 30 00:46:42.223259 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.223198 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" event={"ID":"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e","Type":"ContainerStarted","Data":"79aad0b655fa43822ac7c05622b93d347e13628524535281b729e2da931f9d28"} Oct 30 00:46:42.224989 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.224966 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" event={"ID":"8f5e313a-1534-49cc-bb1c-9ef6a29649e1","Type":"ContainerStarted","Data":"cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6"} Oct 30 00:46:42.278733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.276191 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podStartSLOduration=1.276169569 podStartE2EDuration="1.276169569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:42.275024253 +0000 UTC m=+740.187151417" watchObservedRunningTime="2026-10-30 00:46:42.276169569 +0000 UTC m=+740.188296711" Oct 30 00:46:42.552947 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.552915 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:42.591636 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.591589 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:42.683291 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.683246 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:42.731608 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.731579 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:42.731811 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.731781 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 00:46:42.731867 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:42.731822 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 00:46:43.186730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:43.186632 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:43.311106 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:43.311074 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:43.432294 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:43.432255 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:43.547333 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:43.547300 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:43.731377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:43.731342 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 00:46:43.731549 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:43.731394 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 00:46:44.351147 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.351112 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c"] Oct 30 00:46:44.355382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.355359 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.392118 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.391352 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"openshift-service-ca.crt\"" Oct 30 00:46:44.402552 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.402520 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"kube-root-ca.crt\"" Oct 30 00:46:44.402747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.402585 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"default-dockercfg-82c28\"" Oct 30 00:46:44.402747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.402680 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"simulator-w1-beb8d5-kserve-self-signed-certs\"" Oct 30 00:46:44.404227 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.404208 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c"] Oct 30 00:46:44.411370 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.411345 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1751c25-d9a7-4672-8ca5-e1fafb277608-tls-certs\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.411479 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.411396 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-home\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.411479 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.411423 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-dshm\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.411581 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.411507 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-model-cache\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.411623 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.411598 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-tmp-dir\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.411667 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.411637 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrznk\" (UniqueName: \"kubernetes.io/projected/f1751c25-d9a7-4672-8ca5-e1fafb277608-kube-api-access-xrznk\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.429900 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.429862 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:44.512437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.512387 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-dshm\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.512437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.512426 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-model-cache\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.512695 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.512446 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-tmp-dir\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.512695 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.512474 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrznk\" (UniqueName: \"kubernetes.io/projected/f1751c25-d9a7-4672-8ca5-e1fafb277608-kube-api-access-xrznk\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.512695 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.512586 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1751c25-d9a7-4672-8ca5-e1fafb277608-tls-certs\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.512695 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.512622 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-home\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.512938 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.512877 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-model-cache\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.513014 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.512945 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-tmp-dir\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.513162 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.513142 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-home\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.515460 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.515319 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-dshm\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.515460 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.515415 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1751c25-d9a7-4672-8ca5-e1fafb277608-tls-certs\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.533878 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.533849 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrznk\" (UniqueName: \"kubernetes.io/projected/f1751c25-d9a7-4672-8ca5-e1fafb277608-kube-api-access-xrznk\") pod \"simulator-w1-beb8d5-kserve-6959849856-6wq7c\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.703939 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.703845 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:44.731443 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.731404 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 00:46:44.731565 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:44.731466 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 00:46:45.079133 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:45.079045 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c"] Oct 30 00:46:45.240238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:45.240196 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" event={"ID":"f1751c25-d9a7-4672-8ca5-e1fafb277608","Type":"ContainerStarted","Data":"a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51"} Oct 30 00:46:45.240396 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:45.240245 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" event={"ID":"f1751c25-d9a7-4672-8ca5-e1fafb277608","Type":"ContainerStarted","Data":"f3acb22c0492c9795f16fd231384b0fad891d3070cc805bd3e9058496f7c6920"} Oct 30 00:46:45.280729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:45.280274 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podStartSLOduration=1.280256585 podStartE2EDuration="1.280256585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:45.28007064 +0000 UTC m=+743.192197781" watchObservedRunningTime="2026-10-30 00:46:45.280256585 +0000 UTC m=+743.192383725" Oct 30 00:46:45.330650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:45.330566 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:45.704594 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:45.704510 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:45.731068 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:45.731036 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 00:46:45.731233 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:45.731086 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 00:46:46.590472 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:46.590418 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:46.731460 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:46.731423 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 00:46:46.731911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:46.731476 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 00:46:46.982660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:46.982577 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:47.735998 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:47.735962 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:47.736373 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:47.736280 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:47.736935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:47.736918 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 00:46:48.127861 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:48.127812 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:48.682180 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:48.682144 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:48.819312 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:48.819272 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:48.967174 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:48.967089 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:50.382072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:50.382038 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:51.591090 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:51.591043 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:51.603741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:51.603700 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:52.126673 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:52.126642 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:52.284039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:52.284012 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:46:53.614813 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:53.614774 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:53.758075 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:53.758035 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:54.704557 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:54.704521 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:54.718363 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:54.718335 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:55.300650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.300617 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:46:55.370860 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.370805 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:55.567570 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.567495 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk"] Oct 30 00:46:55.571182 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.571160 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.585563 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.585541 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"openshift-service-ca.crt\"" Oct 30 00:46:55.585694 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.585676 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"simulator-w5-89a676-kserve-self-signed-certs\"" Oct 30 00:46:55.585784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.585761 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"kube-root-ca.crt\"" Oct 30 00:46:55.585834 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.585762 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"default-dockercfg-bpwgn\"" Oct 30 00:46:55.597377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.597352 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk"] Oct 30 00:46:55.616170 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.616131 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-home\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.616292 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.616172 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-model-cache\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.616292 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.616217 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-dshm\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.616380 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.616337 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-tmp-dir\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.616380 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.616365 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64047364-1124-4828-ac28-6151d4bdd2f4-tls-certs\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.616440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.616412 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6jq8\" (UniqueName: \"kubernetes.io/projected/64047364-1124-4828-ac28-6151d4bdd2f4-kube-api-access-v6jq8\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.632326 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.632285 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:55.717178 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.717146 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-home\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.717650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.717193 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-model-cache\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.717650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.717234 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-dshm\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.717650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.717315 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-tmp-dir\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.717650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.717346 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64047364-1124-4828-ac28-6151d4bdd2f4-tls-certs\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.717650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.717402 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6jq8\" (UniqueName: \"kubernetes.io/projected/64047364-1124-4828-ac28-6151d4bdd2f4-kube-api-access-v6jq8\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.717650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.717603 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-home\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.717650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.717627 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-model-cache\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.718046 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.717672 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-tmp-dir\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.719772 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.719745 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-dshm\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.720054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.720036 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64047364-1124-4828-ac28-6151d4bdd2f4-tls-certs\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.740061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.740034 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6jq8\" (UniqueName: \"kubernetes.io/projected/64047364-1124-4828-ac28-6151d4bdd2f4-kube-api-access-v6jq8\") pod \"simulator-w5-89a676-kserve-99658d8dc-wf8sk\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:55.883256 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:55.883169 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:56.184249 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:56.184170 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk"] Oct 30 00:46:56.189209 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:46:56.189165 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64047364_1124_4828_ac28_6151d4bdd2f4.slice/crio-e5f41cd07a812e763f3053e7c7686dfc576e7b4d3a87abc52270eeccb41b4648 WatchSource:0}: Error finding container e5f41cd07a812e763f3053e7c7686dfc576e7b4d3a87abc52270eeccb41b4648: Status 404 returned error can't find the container with id e5f41cd07a812e763f3053e7c7686dfc576e7b4d3a87abc52270eeccb41b4648 Oct 30 00:46:56.295686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:56.295648 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" event={"ID":"64047364-1124-4828-ac28-6151d4bdd2f4","Type":"ContainerStarted","Data":"9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582"} Oct 30 00:46:56.295873 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:56.295696 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" event={"ID":"64047364-1124-4828-ac28-6151d4bdd2f4","Type":"ContainerStarted","Data":"e5f41cd07a812e763f3053e7c7686dfc576e7b4d3a87abc52270eeccb41b4648"} Oct 30 00:46:56.335718 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:56.335657 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podStartSLOduration=1.335638595 podStartE2EDuration="1.335638595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:56.334497866 +0000 UTC m=+754.246625026" watchObservedRunningTime="2026-10-30 00:46:56.335638595 +0000 UTC m=+754.247765733" Oct 30 00:46:56.883603 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:56.883568 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:46:57.122369 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:57.122328 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:57.437113 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:57.437080 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:58.799931 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:46:58.799899 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:00.447336 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:00.447302 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:00.760206 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:47:00.760116 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:47:02.329247 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:02.329137 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:03.976348 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:03.976315 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:04.714127 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.714094 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx"] Oct 30 00:47:04.718069 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.718046 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.742494 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.742470 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"default-dockercfg-wmkj4\"" Oct 30 00:47:04.742494 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.742479 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"simulator-w0-2b4105-kserve-self-signed-certs\"" Oct 30 00:47:04.742494 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.742489 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"openshift-service-ca.crt\"" Oct 30 00:47:04.742777 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.742497 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"kube-root-ca.crt\"" Oct 30 00:47:04.745251 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.745227 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx"] Oct 30 00:47:04.774429 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.774393 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:04.803514 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.803457 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-dshm\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.803698 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.803518 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731830bd-e885-4e72-a650-6682c5aa029b-tls-certs\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.803698 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.803590 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-tmp-dir\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.803698 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.803651 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-home\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.803698 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.803691 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpx9\" (UniqueName: \"kubernetes.io/projected/731830bd-e885-4e72-a650-6682c5aa029b-kube-api-access-8dpx9\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.803947 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.803754 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-model-cache\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.904411 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.904371 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-tmp-dir\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.904590 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.904446 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-home\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.904590 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.904486 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpx9\" (UniqueName: \"kubernetes.io/projected/731830bd-e885-4e72-a650-6682c5aa029b-kube-api-access-8dpx9\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.904590 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.904524 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-model-cache\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.904769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.904591 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-dshm\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.904769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.904623 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731830bd-e885-4e72-a650-6682c5aa029b-tls-certs\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.904912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.904889 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-tmp-dir\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.904988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.904966 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-home\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.905173 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.905151 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-model-cache\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.907609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.907583 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731830bd-e885-4e72-a650-6682c5aa029b-tls-certs\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.907756 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.907692 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-dshm\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:04.926995 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:04.926959 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpx9\" (UniqueName: \"kubernetes.io/projected/731830bd-e885-4e72-a650-6682c5aa029b-kube-api-access-8dpx9\") pod \"simulator-w0-2b4105-kserve-54755c796c-qwwvx\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:05.031635 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:05.031546 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:05.329084 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:05.329045 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx"] Oct 30 00:47:05.333248 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:47:05.333215 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod731830bd_e885_4e72_a650_6682c5aa029b.slice/crio-5276f9ee636ec619da1a37f208b6e9f9cdd0c28bdecef29b4121c89d0be66953 WatchSource:0}: Error finding container 5276f9ee636ec619da1a37f208b6e9f9cdd0c28bdecef29b4121c89d0be66953: Status 404 returned error can't find the container with id 5276f9ee636ec619da1a37f208b6e9f9cdd0c28bdecef29b4121c89d0be66953 Oct 30 00:47:05.341116 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:05.341086 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" event={"ID":"731830bd-e885-4e72-a650-6682c5aa029b","Type":"ContainerStarted","Data":"5276f9ee636ec619da1a37f208b6e9f9cdd0c28bdecef29b4121c89d0be66953"} Oct 30 00:47:05.623106 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:05.622992 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:05.883537 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:05.883438 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:47:05.896185 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:05.896144 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:47:06.349601 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:06.349568 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" event={"ID":"731830bd-e885-4e72-a650-6682c5aa029b","Type":"ContainerStarted","Data":"c6e72baa4e914436896014d83e9eb04ccdd0d5dd0cd7a6560b33609dc6c67a34"} Oct 30 00:47:06.360464 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:06.360442 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:47:06.386921 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:06.386872 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podStartSLOduration=2.386852463 podStartE2EDuration="2.386852463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:06.386269373 +0000 UTC m=+764.298396513" watchObservedRunningTime="2026-10-30 00:47:06.386852463 +0000 UTC m=+764.298979609" Oct 30 00:47:07.032071 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:07.032040 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:07.313019 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:07.312988 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:08.979306 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:08.979267 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:10.013843 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:10.013812 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:10.634340 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:10.634310 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:12.429407 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:12.429373 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:13.375135 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:13.375098 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:14.041391 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:14.041360 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:15.031870 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:15.031782 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:15.044070 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:15.044044 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:15.414191 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:15.414165 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:47:15.633299 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:15.633265 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:17.338053 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:17.338018 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:19.008080 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:19.008044 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:20.766205 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:20.766176 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:21.163042 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:21.162977 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:23.083069 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:23.083003 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:24.862074 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:24.862022 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:26.636494 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:26.636461 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:28.273201 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:28.273166 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:29.911598 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:29.911553 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:30.605426 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.605389 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:30.715813 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.715783 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:30.764441 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:47:30.764403 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:47:30.909884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.909792 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt"] Oct 30 00:47:30.914984 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.914954 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:30.938278 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.938249 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"premium-w4-6add35-kserve-self-signed-certs\"" Oct 30 00:47:30.940149 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.940122 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt"] Oct 30 00:47:30.964611 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.964574 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-home\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:30.964611 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.964615 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-model-cache\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:30.964853 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.964646 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-dshm\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:30.964853 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.964746 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tmp-dir\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:30.964853 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.964780 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwvmp\" (UniqueName: \"kubernetes.io/projected/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-kube-api-access-xwvmp\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:30.964853 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.964820 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tls-certs\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:30.976234 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:30.976202 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:31.066236 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.066197 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-dshm\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.066440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.066248 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tmp-dir\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.066440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.066385 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwvmp\" (UniqueName: \"kubernetes.io/projected/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-kube-api-access-xwvmp\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.066562 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.066469 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tls-certs\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.066619 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.066569 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-home\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.066619 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.066599 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-model-cache\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.066842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.066795 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tmp-dir\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.066983 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.066952 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-model-cache\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.067268 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.067136 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-home\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.069064 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.069035 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-dshm\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.069732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.069698 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tls-certs\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.088182 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.088151 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwvmp\" (UniqueName: \"kubernetes.io/projected/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-kube-api-access-xwvmp\") pod \"premium-w4-6add35-kserve-d54c49f65-vhnwt\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.229019 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.228926 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:31.532520 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.532446 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt"] Oct 30 00:47:31.536136 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:47:31.536091 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7ed8efc_1cab_412b_a7d7_9ba1899d6ce9.slice/crio-0d3b0a5b9b271f5a75d2fad1b1f6571a4b068d3908d9baece833b9510dff5950 WatchSource:0}: Error finding container 0d3b0a5b9b271f5a75d2fad1b1f6571a4b068d3908d9baece833b9510dff5950: Status 404 returned error can't find the container with id 0d3b0a5b9b271f5a75d2fad1b1f6571a4b068d3908d9baece833b9510dff5950 Oct 30 00:47:31.639840 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:31.639809 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:32.482677 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:32.482637 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" event={"ID":"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9","Type":"ContainerStarted","Data":"d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef"} Oct 30 00:47:32.483116 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:32.482689 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" event={"ID":"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9","Type":"ContainerStarted","Data":"0d3b0a5b9b271f5a75d2fad1b1f6571a4b068d3908d9baece833b9510dff5950"} Oct 30 00:47:32.521048 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:32.520991 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podStartSLOduration=2.520972748 podStartE2EDuration="2.520972748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:32.520152554 +0000 UTC m=+790.432279696" watchObservedRunningTime="2026-10-30 00:47:32.520972748 +0000 UTC m=+790.433099891" Oct 30 00:47:33.136870 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.136662 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q"] Oct 30 00:47:33.140349 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.140323 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.165239 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.165212 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"premium-w1-beb8d5-kserve-self-signed-certs\"" Oct 30 00:47:33.168047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.168006 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q"] Oct 30 00:47:33.217249 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.217220 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:33.229090 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.229058 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:33.287117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.287083 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5lpk\" (UniqueName: \"kubernetes.io/projected/d8fabd8b-f688-4146-bc0d-742d7c1953ff-kube-api-access-m5lpk\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.287331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.287135 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-home\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.287331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.287207 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tls-certs\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.287331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.287228 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-dshm\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.287331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.287319 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-model-cache\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.287556 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.287336 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tmp-dir\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.327366 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.327321 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:33.388438 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.388338 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-model-cache\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.388438 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.388380 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tmp-dir\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.388438 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.388420 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5lpk\" (UniqueName: \"kubernetes.io/projected/d8fabd8b-f688-4146-bc0d-742d7c1953ff-kube-api-access-m5lpk\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.388753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.388467 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-home\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.388753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.388497 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tls-certs\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.388753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.388521 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-dshm\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.388919 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.388866 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-model-cache\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.388976 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.388918 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tmp-dir\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.389049 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.389030 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-home\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.391076 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.391051 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-dshm\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.391491 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.391475 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tls-certs\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.410241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.410208 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5lpk\" (UniqueName: \"kubernetes.io/projected/d8fabd8b-f688-4146-bc0d-742d7c1953ff-kube-api-access-m5lpk\") pod \"premium-w1-beb8d5-kserve-764dff75cc-5ts8q\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.456115 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.455964 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:33.757779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:33.757673 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q"] Oct 30 00:47:33.761947 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:47:33.761907 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8fabd8b_f688_4146_bc0d_742d7c1953ff.slice/crio-bad8d07937ccdf054c3fc0887b85b4b0e6c20b09e081a66a4968ddc3fb23ba54 WatchSource:0}: Error finding container bad8d07937ccdf054c3fc0887b85b4b0e6c20b09e081a66a4968ddc3fb23ba54: Status 404 returned error can't find the container with id bad8d07937ccdf054c3fc0887b85b4b0e6c20b09e081a66a4968ddc3fb23ba54 Oct 30 00:47:34.493857 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:34.493807 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" event={"ID":"d8fabd8b-f688-4146-bc0d-742d7c1953ff","Type":"ContainerStarted","Data":"c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60"} Oct 30 00:47:34.494060 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:34.493868 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" event={"ID":"d8fabd8b-f688-4146-bc0d-742d7c1953ff","Type":"ContainerStarted","Data":"bad8d07937ccdf054c3fc0887b85b4b0e6c20b09e081a66a4968ddc3fb23ba54"} Oct 30 00:47:34.532550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:34.532499 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podStartSLOduration=1.532484695 podStartE2EDuration="1.532484695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:34.530908558 +0000 UTC m=+792.443035726" watchObservedRunningTime="2026-10-30 00:47:34.532484695 +0000 UTC m=+792.444611837" Oct 30 00:47:35.028501 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:35.028470 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:35.456277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:35.456242 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:36.723785 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:36.723746 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:38.514309 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:38.514269 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:40.066404 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:40.066358 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:41.229554 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:41.229526 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:41.241852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:41.241814 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:41.537435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:41.537407 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:47:41.800793 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:41.800674 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:41.909431 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:41.909395 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:42.935202 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:42.935170 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28"] Oct 30 00:47:42.939038 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:42.939010 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:42.952254 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:42.952230 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"default-dockercfg-jj8kk\"" Oct 30 00:47:42.962767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:42.962743 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"premium-w3-b3593a-kserve-self-signed-certs\"" Oct 30 00:47:42.962948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:42.962803 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"kube-root-ca.crt\"" Oct 30 00:47:42.962948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:42.962875 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"openshift-service-ca.crt\"" Oct 30 00:47:42.965688 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:42.965660 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28"] Oct 30 00:47:42.993072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:42.993024 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:43.083007 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.082976 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-home\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.083186 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.083014 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tls-certs\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.083186 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.083156 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmm2k\" (UniqueName: \"kubernetes.io/projected/1af0882d-f4f4-4613-b90f-42ef0b01efc6-kube-api-access-wmm2k\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.083276 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.083259 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-dshm\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.083326 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.083317 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-model-cache\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.083380 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.083367 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tmp-dir\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.184246 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.184200 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmm2k\" (UniqueName: \"kubernetes.io/projected/1af0882d-f4f4-4613-b90f-42ef0b01efc6-kube-api-access-wmm2k\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.184423 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.184300 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-dshm\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.184423 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.184373 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-model-cache\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.184423 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.184407 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tmp-dir\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.184616 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.184482 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-home\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.184616 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.184515 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tls-certs\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.184906 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.184877 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-model-cache\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.185017 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.184945 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-home\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.185125 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.185108 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tmp-dir\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.187189 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.187127 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-dshm\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.188082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.187754 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tls-certs\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.205788 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.205752 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmm2k\" (UniqueName: \"kubernetes.io/projected/1af0882d-f4f4-4613-b90f-42ef0b01efc6-kube-api-access-wmm2k\") pod \"premium-w3-b3593a-kserve-6c9db7bd68-xwp28\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.255718 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.255671 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:43.456419 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.456318 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:43.474040 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.474002 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:43.529064 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.529019 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:43.549090 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.549061 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:47:43.557383 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:43.557354 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28"] Oct 30 00:47:43.561473 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:47:43.561446 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1af0882d_f4f4_4613_b90f_42ef0b01efc6.slice/crio-3619e45bd90c948e89bf45325ea24b3a77e97f3881ad3c466d1e18313f5413bc WatchSource:0}: Error finding container 3619e45bd90c948e89bf45325ea24b3a77e97f3881ad3c466d1e18313f5413bc: Status 404 returned error can't find the container with id 3619e45bd90c948e89bf45325ea24b3a77e97f3881ad3c466d1e18313f5413bc Oct 30 00:47:44.544794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:44.544751 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" event={"ID":"1af0882d-f4f4-4613-b90f-42ef0b01efc6","Type":"ContainerStarted","Data":"32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b"} Oct 30 00:47:44.545301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:44.544822 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" event={"ID":"1af0882d-f4f4-4613-b90f-42ef0b01efc6","Type":"ContainerStarted","Data":"3619e45bd90c948e89bf45325ea24b3a77e97f3881ad3c466d1e18313f5413bc"} Oct 30 00:47:44.583026 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:44.582956 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podStartSLOduration=2.582937593 podStartE2EDuration="2.582937593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:44.581024035 +0000 UTC m=+802.493151175" watchObservedRunningTime="2026-10-30 00:47:44.582937593 +0000 UTC m=+802.495064735" Oct 30 00:47:45.229766 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:45.229732 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:45.256309 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:45.256270 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:45.948260 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:45.948221 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:46.854239 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:46.854210 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:48.621967 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:48.621929 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:50.312025 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:50.311989 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:52.038588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:52.038554 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:53.255871 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:53.255818 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:53.268684 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:53.268658 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:53.603545 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:53.603514 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:47:53.965741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:53.965627 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:55.813061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:55.813024 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:57.513408 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:57.513371 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:59.324184 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:47:59.324151 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:00.758245 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:48:00.758200 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:48:00.960354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:00.960320 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:02.910099 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:02.910068 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:03.356234 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.356203 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st"] Oct 30 00:48:03.360436 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.360412 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.365725 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.365687 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.365868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.365794 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl8lb\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-kube-api-access-sl8lb\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.365868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.365840 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.365990 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.365900 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.365990 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.365958 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.365990 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.365979 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.366136 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.366047 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.366136 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.366091 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.366232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.366208 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.373547 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.373529 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6add35-openshift-default-dockercfg-5zrdb\"" Oct 30 00:48:03.390488 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.390459 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st"] Oct 30 00:48:03.468779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.468749 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.468972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.468811 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.468972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.468841 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.468972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.468862 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.468972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.468893 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.468972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.468931 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.469231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.468981 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.469231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.469031 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.469231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.469072 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8lb\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-kube-api-access-sl8lb\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.469440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.469227 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.469440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.469293 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.469910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.469880 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.470009 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.469986 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.470072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.470034 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.471840 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.471775 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.472064 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.472044 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.490859 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.490835 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.490986 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.490864 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl8lb\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-kube-api-access-sl8lb\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:03.675867 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:03.675771 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:04.034656 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:48:04.034629 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5218673d_d5e7_40c3_ae83_3e5897e3ed12.slice/crio-fdea1aa6bd6fd181df4ef9c9dbc23d6847f5a82a0ae8c160ca6505267ae67fd4 WatchSource:0}: Error finding container fdea1aa6bd6fd181df4ef9c9dbc23d6847f5a82a0ae8c160ca6505267ae67fd4: Status 404 returned error can't find the container with id fdea1aa6bd6fd181df4ef9c9dbc23d6847f5a82a0ae8c160ca6505267ae67fd4 Oct 30 00:48:04.037013 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:04.036978 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:04.037102 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:04.037055 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:04.037102 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:04.037085 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:04.041379 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:04.041352 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st"] Oct 30 00:48:04.648974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:04.648932 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" event={"ID":"5218673d-d5e7-40c3-ae83-3e5897e3ed12","Type":"ContainerStarted","Data":"0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab"} Oct 30 00:48:04.649158 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:04.648995 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" event={"ID":"5218673d-d5e7-40c3-ae83-3e5897e3ed12","Type":"ContainerStarted","Data":"fdea1aa6bd6fd181df4ef9c9dbc23d6847f5a82a0ae8c160ca6505267ae67fd4"} Oct 30 00:48:04.675908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:04.675878 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:04.712848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:04.712666 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" podStartSLOduration=1.7126488819999999 podStartE2EDuration="1.712648882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:04.700552464 +0000 UTC m=+822.612679605" watchObservedRunningTime="2026-10-30 00:48:04.712648882 +0000 UTC m=+822.624776026" Oct 30 00:48:04.839508 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:04.839471 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:05.680734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:05.680443 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:48:05.680734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:05.680513 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:48:06.465266 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:06.465191 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:06.680137 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:06.680097 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:48:06.680462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:06.680432 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:48:07.679733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:07.679137 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:48:07.679733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:07.679201 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:48:08.241828 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:08.241795 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:08.681735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:08.678162 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:48:08.681735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:08.678241 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:48:09.680162 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:09.680122 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:48:09.680480 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:09.680445 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:48:09.880735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:09.877956 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:10.676311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:10.676277 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:48:10.676498 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:10.676329 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:48:10.946537 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:10.946452 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn"] Oct 30 00:48:10.950291 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:10.950271 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:10.963381 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:10.963354 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:48:10.973586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:10.973556 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:48:10.976802 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:10.976765 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn"] Oct 30 00:48:11.052161 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.052123 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3eb6a08-2lzmn\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.052342 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.052163 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qlg\" (UniqueName: \"kubernetes.io/projected/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-kube-api-access-w2qlg\") pod \"maas-api-revoke-sub-b3eb6a08-2lzmn\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.052342 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.052194 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3eb6a08-2lzmn\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.153241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.153201 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3eb6a08-2lzmn\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.153241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.153242 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qlg\" (UniqueName: \"kubernetes.io/projected/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-kube-api-access-w2qlg\") pod \"maas-api-revoke-sub-b3eb6a08-2lzmn\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.153511 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.153264 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3eb6a08-2lzmn\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.153985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.153966 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3eb6a08-2lzmn\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.155657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.155639 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3eb6a08-2lzmn\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.174767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.174731 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qlg\" (UniqueName: \"kubernetes.io/projected/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-kube-api-access-w2qlg\") pod \"maas-api-revoke-sub-b3eb6a08-2lzmn\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.266458 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.266378 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:11.529526 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.529434 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:11.570823 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.570783 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn"] Oct 30 00:48:11.681414 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.681385 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:11.681698 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.681677 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:11.682450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.682435 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 00:48:11.707230 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:11.707200 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" event={"ID":"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3","Type":"ContainerStarted","Data":"632698e3d41f16c05261b53a11a96b90ebfc6b76f004eba0cfd857d6b98f6288"} Oct 30 00:48:13.232383 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:13.232354 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:13.498257 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:13.498163 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:14.724674 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:14.724638 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" event={"ID":"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3","Type":"ContainerStarted","Data":"b3afc39d7d83659e2867d60790152f0d0a27ee0fb30514046a63e391972b6360"} Oct 30 00:48:14.766755 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:14.766680 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" podStartSLOduration=2.100482581 podStartE2EDuration="4.766665552s" totalImagesPullingTime="2.666182971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:14.764991881 +0000 UTC m=+832.677119051" watchObservedRunningTime="2026-10-30 00:48:14.766665552 +0000 UTC m=+832.678792693" Oct 30 00:48:14.819913 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:14.819884 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:15.731041 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:15.731001 4517 generic.go:356] "Generic (PLEG): container finished" podID="21b7c7cf-b72c-45b6-b297-f07e1ace4ef3" containerID="b3afc39d7d83659e2867d60790152f0d0a27ee0fb30514046a63e391972b6360" exitCode=0 Oct 30 00:48:15.731492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:15.731068 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" event={"ID":"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3","Type":"ContainerDied","Data":"b3afc39d7d83659e2867d60790152f0d0a27ee0fb30514046a63e391972b6360"} Oct 30 00:48:16.463746 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:16.463697 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:16.915742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:16.915696 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:17.017286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.017199 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-openshift-ca\") pod \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " Oct 30 00:48:17.017286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.017266 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-kube-api-access-w2qlg\" (UniqueName: \"kubernetes.io/projected/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-kube-api-access-w2qlg\") pod \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " Oct 30 00:48:17.017468 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.017293 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-xks-ca\") pod \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\" (UID: \"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3\") " Oct 30 00:48:17.017656 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.017630 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-openshift-ca" pod "21b7c7cf-b72c-45b6-b297-f07e1ace4ef3" (UID: "21b7c7cf-b72c-45b6-b297-f07e1ace4ef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:17.017766 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.017751 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:48:17.019911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.019881 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-xks-ca" pod "21b7c7cf-b72c-45b6-b297-f07e1ace4ef3" (UID: "21b7c7cf-b72c-45b6-b297-f07e1ace4ef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:17.020068 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.020003 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-kube-api-access-w2qlg" pod "21b7c7cf-b72c-45b6-b297-f07e1ace4ef3" (UID: "21b7c7cf-b72c-45b6-b297-f07e1ace4ef3"). InnerVolumeSpecName "kube-api-access-w2qlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:17.118476 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.118423 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qlg\" (UniqueName: \"kubernetes.io/projected/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-kube-api-access-w2qlg\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:48:17.118476 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.118471 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:48:17.742821 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.742787 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" Oct 30 00:48:17.743008 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.742786 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn" event={"ID":"21b7c7cf-b72c-45b6-b297-f07e1ace4ef3","Type":"ContainerDied","Data":"632698e3d41f16c05261b53a11a96b90ebfc6b76f004eba0cfd857d6b98f6288"} Oct 30 00:48:17.743008 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:17.742914 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="632698e3d41f16c05261b53a11a96b90ebfc6b76f004eba0cfd857d6b98f6288" Oct 30 00:48:18.251400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.248639 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm"] Oct 30 00:48:18.251400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.249482 4517 cpu_manager.go:414] "containerMap: removing container" podUID="21b7c7cf-b72c-45b6-b297-f07e1ace4ef3" containerName="revoke-keys" Oct 30 00:48:18.251400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.249501 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b7c7cf-b72c-45b6-b297-f07e1ace4ef3" containerName="revoke-keys" Oct 30 00:48:18.251400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.249652 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b7c7cf-b72c-45b6-b297-f07e1ace4ef3" containerName="revoke-keys" Oct 30 00:48:18.253914 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.253886 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.271896 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.271866 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-beb8d5-openshift-default-dockercfg-kjcx7\"" Oct 30 00:48:18.287131 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.287083 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm"] Oct 30 00:48:18.330762 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.330667 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.330911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.330799 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.330911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.330843 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.330911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.330881 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.331039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.330914 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.331081 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.331046 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntvpq\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-kube-api-access-ntvpq\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.331132 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.331084 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2479194-1584-4a8f-a67a-bd90fbb3596b-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.331264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.331228 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.331398 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.331295 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.338134 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.338106 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:18.432600 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.432562 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.432795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.432623 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.432795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.432765 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.432892 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.432796 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.432892 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.432823 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.432892 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.432858 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.433044 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.432892 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.433044 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.432939 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntvpq\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-kube-api-access-ntvpq\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.433044 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.432968 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2479194-1584-4a8f-a67a-bd90fbb3596b-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.433204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.433174 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.433891 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.433856 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.434029 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.433901 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.434189 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.434161 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.434400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.434376 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2479194-1584-4a8f-a67a-bd90fbb3596b-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.435480 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.435449 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.436141 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.436122 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.455486 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.455462 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.455852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.455833 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntvpq\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-kube-api-access-ntvpq\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.522810 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.522698 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r"] Oct 30 00:48:18.527515 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.527492 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.551217 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.551185 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r"] Oct 30 00:48:18.575314 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.575270 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:18.628378 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.628344 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b3593a-openshift-default-dockercfg-knq67\"" Oct 30 00:48:18.635584 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.635556 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfvz\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-kube-api-access-zmfvz\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.635806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.635783 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.636006 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.635991 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.636231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.636211 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.636415 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.636397 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.637272 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.636799 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.637272 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.636843 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.637272 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.636874 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.637272 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.636898 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.737941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.737907 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.737941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.737948 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.737970 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.737989 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738198 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738378 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738329 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfvz\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-kube-api-access-zmfvz\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738411 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738523 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738483 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738523 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738506 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738696 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738665 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738699 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738750 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738791 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.738994 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.738918 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.741125 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.741100 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.741211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.741123 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.761055 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.761027 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.761206 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.761181 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfvz\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-kube-api-access-zmfvz\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.843191 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.843160 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:18.910384 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.910342 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm"] Oct 30 00:48:18.911449 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:48:18.911415 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2479194_1584_4a8f_a67a_bd90fbb3596b.slice/crio-7d5904ebca1aef138e403e05a318539aa3ebf8e849340c38fe7bc333ae4e4dce WatchSource:0}: Error finding container 7d5904ebca1aef138e403e05a318539aa3ebf8e849340c38fe7bc333ae4e4dce: Status 404 returned error can't find the container with id 7d5904ebca1aef138e403e05a318539aa3ebf8e849340c38fe7bc333ae4e4dce Oct 30 00:48:18.917329 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.917296 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:18.917410 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.917364 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:18.917410 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:18.917395 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:19.156679 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.156595 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r"] Oct 30 00:48:19.164646 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:48:19.164611 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920fe4a0_48c7_4b43_a746_0543f8b3e9bf.slice/crio-d8d91d613c9aeb253a4b43e8aeb43391e6b03c828585ca9ed40d353b8b57b961 WatchSource:0}: Error finding container d8d91d613c9aeb253a4b43e8aeb43391e6b03c828585ca9ed40d353b8b57b961: Status 404 returned error can't find the container with id d8d91d613c9aeb253a4b43e8aeb43391e6b03c828585ca9ed40d353b8b57b961 Oct 30 00:48:19.168030 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.167993 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:19.168104 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.168070 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:19.168149 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.168108 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:19.754050 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.754016 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" event={"ID":"920fe4a0-48c7-4b43-a746-0543f8b3e9bf","Type":"ContainerStarted","Data":"b02c16a9c48ba52328b0103ed68be0c3a47e1467925c27a3180bbe89768882d2"} Oct 30 00:48:19.754527 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.754077 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" event={"ID":"920fe4a0-48c7-4b43-a746-0543f8b3e9bf","Type":"ContainerStarted","Data":"d8d91d613c9aeb253a4b43e8aeb43391e6b03c828585ca9ed40d353b8b57b961"} Oct 30 00:48:19.755521 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.755498 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" event={"ID":"d2479194-1584-4a8f-a67a-bd90fbb3596b","Type":"ContainerStarted","Data":"9a72503bc7c14cc42078c0f02bf4174a37eda035257e3d85f6087647ac8d257e"} Oct 30 00:48:19.755615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.755546 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" event={"ID":"d2479194-1584-4a8f-a67a-bd90fbb3596b","Type":"ContainerStarted","Data":"7d5904ebca1aef138e403e05a318539aa3ebf8e849340c38fe7bc333ae4e4dce"} Oct 30 00:48:19.840665 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.840613 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" podStartSLOduration=1.840600139 podStartE2EDuration="1.840600139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:19.828587588 +0000 UTC m=+837.740714725" watchObservedRunningTime="2026-10-30 00:48:19.840600139 +0000 UTC m=+837.752727286" Oct 30 00:48:19.843788 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:19.843767 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:20.078237 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:20.075188 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:20.576113 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:20.576078 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:20.593801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:20.593755 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:48:20.594083 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:20.594052 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:48:20.845152 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:20.845059 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:48:20.845152 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:20.845115 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:48:21.576177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:21.576138 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:48:21.576474 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:21.576444 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:48:21.844630 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:21.844255 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:48:21.844630 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:21.844302 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:48:22.156201 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:22.156130 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:22.578617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:22.578148 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:48:22.578617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:22.578213 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:48:22.844241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:22.844137 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:48:22.844241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:22.844199 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:48:23.578740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:23.576210 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:48:23.578740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:23.576283 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:48:23.844569 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:23.844486 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:48:23.844569 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:23.844532 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:48:24.100794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:24.099736 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:24.585737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:24.585294 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:48:24.585737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:24.585390 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:48:24.844592 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:24.844501 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:48:24.844797 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:24.844587 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:48:25.576558 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.576518 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:48:25.576806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.576592 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:48:25.634041 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.633988 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" podStartSLOduration=7.633973997 podStartE2EDuration="7.633973997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:19.889827025 +0000 UTC m=+837.801954169" watchObservedRunningTime="2026-10-30 00:48:25.633973997 +0000 UTC m=+843.546101137" Oct 30 00:48:25.634528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.634467 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp"] Oct 30 00:48:25.639140 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.639118 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.661640 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.655052 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:48:25.672743 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.669001 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp"] Oct 30 00:48:25.673410 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.673383 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:48:25.755678 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.755649 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21946d9d-26pjp\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.755857 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.755835 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21946d9d-26pjp\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.755918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.755898 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdzg\" (UniqueName: \"kubernetes.io/projected/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-kube-api-access-5vdzg\") pod \"maas-api-revoke-sub-21946d9d-26pjp\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.843901 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.843808 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:48:25.843901 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.843880 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:48:25.857507 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.857459 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21946d9d-26pjp\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.857670 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.857550 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdzg\" (UniqueName: \"kubernetes.io/projected/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-kube-api-access-5vdzg\") pod \"maas-api-revoke-sub-21946d9d-26pjp\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.857670 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.857591 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21946d9d-26pjp\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.859024 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.858994 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21946d9d-26pjp\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.860547 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.860522 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21946d9d-26pjp\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.880321 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.880294 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdzg\" (UniqueName: \"kubernetes.io/projected/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-kube-api-access-5vdzg\") pod \"maas-api-revoke-sub-21946d9d-26pjp\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:25.987051 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:25.987008 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:26.099456 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.098496 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:26.327550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.327511 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp"] Oct 30 00:48:26.331687 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:48:26.331662 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc8759e_efb0_4a64_94ae_48ccf2a8c798.slice/crio-af11d24edb309460affa6cb4e218ab456015b468dc3ac826d3a685f9d1747ad8 WatchSource:0}: Error finding container af11d24edb309460affa6cb4e218ab456015b468dc3ac826d3a685f9d1747ad8: Status 404 returned error can't find the container with id af11d24edb309460affa6cb4e218ab456015b468dc3ac826d3a685f9d1747ad8 Oct 30 00:48:26.580922 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.580893 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:26.581259 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.581241 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:26.582080 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.582063 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 00:48:26.831649 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.831609 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" event={"ID":"3fc8759e-efb0-4a64-94ae-48ccf2a8c798","Type":"ContainerStarted","Data":"2b72bc2e6aaa2ffc5b1fb3f107fde7c30b1798148bd43b732c6098fd72eb94e0"} Oct 30 00:48:26.832091 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.831676 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" event={"ID":"3fc8759e-efb0-4a64-94ae-48ccf2a8c798","Type":"ContainerStarted","Data":"af11d24edb309460affa6cb4e218ab456015b468dc3ac826d3a685f9d1747ad8"} Oct 30 00:48:26.849148 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.849123 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:26.849415 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.849397 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:26.850128 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.850112 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 00:48:26.869672 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:26.869629 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" podStartSLOduration=1.869616057 podStartE2EDuration="1.869616057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:26.868252642 +0000 UTC m=+844.780379784" watchObservedRunningTime="2026-10-30 00:48:26.869616057 +0000 UTC m=+844.781743198" Oct 30 00:48:27.838242 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:27.838211 4517 generic.go:356] "Generic (PLEG): container finished" podID="3fc8759e-efb0-4a64-94ae-48ccf2a8c798" containerID="2b72bc2e6aaa2ffc5b1fb3f107fde7c30b1798148bd43b732c6098fd72eb94e0" exitCode=0 Oct 30 00:48:27.838751 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:27.838301 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" event={"ID":"3fc8759e-efb0-4a64-94ae-48ccf2a8c798","Type":"ContainerDied","Data":"2b72bc2e6aaa2ffc5b1fb3f107fde7c30b1798148bd43b732c6098fd72eb94e0"} Oct 30 00:48:27.862211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:27.862167 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:28.990695 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:28.990664 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:29.093663 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.093617 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-openshift-ca\") pod \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " Oct 30 00:48:29.093864 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.093729 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-xks-ca\") pod \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " Oct 30 00:48:29.093864 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.093768 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-kube-api-access-5vdzg\" (UniqueName: \"kubernetes.io/projected/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-kube-api-access-5vdzg\") pod \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\" (UID: \"3fc8759e-efb0-4a64-94ae-48ccf2a8c798\") " Oct 30 00:48:29.094116 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.094088 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-openshift-ca" pod "3fc8759e-efb0-4a64-94ae-48ccf2a8c798" (UID: "3fc8759e-efb0-4a64-94ae-48ccf2a8c798"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:29.096136 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.096063 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-kube-api-access-5vdzg" pod "3fc8759e-efb0-4a64-94ae-48ccf2a8c798" (UID: "3fc8759e-efb0-4a64-94ae-48ccf2a8c798"). InnerVolumeSpecName "kube-api-access-5vdzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:29.096249 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.096129 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-xks-ca" pod "3fc8759e-efb0-4a64-94ae-48ccf2a8c798" (UID: "3fc8759e-efb0-4a64-94ae-48ccf2a8c798"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:29.195009 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.194954 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:48:29.195009 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.195007 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:48:29.195231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.195022 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vdzg\" (UniqueName: \"kubernetes.io/projected/3fc8759e-efb0-4a64-94ae-48ccf2a8c798-kube-api-access-5vdzg\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:48:29.491972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.491878 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:29.850211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.850169 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" event={"ID":"3fc8759e-efb0-4a64-94ae-48ccf2a8c798","Type":"ContainerDied","Data":"af11d24edb309460affa6cb4e218ab456015b468dc3ac826d3a685f9d1747ad8"} Oct 30 00:48:29.850395 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.850228 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af11d24edb309460affa6cb4e218ab456015b468dc3ac826d3a685f9d1747ad8" Oct 30 00:48:29.850395 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:29.850185 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp" Oct 30 00:48:30.758285 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:48:30.758241 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:48:31.140409 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:31.140373 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:33.245914 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:33.245843 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:34.871765 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:34.871702 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:36.914023 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:36.913989 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:38.532161 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:38.532120 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:38.646517 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:38.646478 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:40.328668 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:40.328628 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:42.039583 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:42.039551 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:43.646237 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:43.646199 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:45.522150 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:45.522109 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:47.244475 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:47.244436 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:47.415920 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:47.415883 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:48.921809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:48.921769 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:50.690395 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:50.690352 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:51.367615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.367586 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx"] Oct 30 00:48:51.367999 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.367986 4517 cpu_manager.go:414] "containerMap: removing container" podUID="3fc8759e-efb0-4a64-94ae-48ccf2a8c798" containerName="revoke-keys" Oct 30 00:48:51.368053 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.368000 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc8759e-efb0-4a64-94ae-48ccf2a8c798" containerName="revoke-keys" Oct 30 00:48:51.368088 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.368082 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc8759e-efb0-4a64-94ae-48ccf2a8c798" containerName="revoke-keys" Oct 30 00:48:51.374464 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.374444 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.400648 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.400623 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx"] Oct 30 00:48:51.561516 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.561480 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.561741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.561537 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.561741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.561626 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.561741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.561674 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.561922 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.561784 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grr6\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-kube-api-access-5grr6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.561922 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.561842 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.561991 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.561925 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.562032 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.561988 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.562076 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.562026 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.662739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.662615 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.662739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.662663 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.662985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.662755 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.662985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.662812 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.662985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.662853 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.662985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.662900 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.662985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.662939 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grr6\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-kube-api-access-5grr6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.662985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.662979 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.663286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.663029 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.663286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.663104 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.663286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.663254 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.663478 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.663455 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.663825 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.663678 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.664277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.664236 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.665793 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.665774 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.666015 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.665996 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.684528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.684493 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.684619 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.684584 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grr6\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-kube-api-access-5grr6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:51.689391 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:51.689373 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:52.013042 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:52.011069 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx"] Oct 30 00:48:52.023892 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:48:52.023857 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8a756b_d0ec_4fec_952c_4ae5e28b54f8.slice/crio-330bffd75fdd4e4646bfa32aec6d9276664adeee910be3452d02717b3e60f032 WatchSource:0}: Error finding container 330bffd75fdd4e4646bfa32aec6d9276664adeee910be3452d02717b3e60f032: Status 404 returned error can't find the container with id 330bffd75fdd4e4646bfa32aec6d9276664adeee910be3452d02717b3e60f032 Oct 30 00:48:52.026313 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:52.026278 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:52.026368 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:52.026350 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:52.026411 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:52.026377 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:48:52.064235 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:52.064203 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" event={"ID":"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8","Type":"ContainerStarted","Data":"330bffd75fdd4e4646bfa32aec6d9276664adeee910be3452d02717b3e60f032"} Oct 30 00:48:52.745848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:52.745810 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:53.088947 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:53.088906 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" event={"ID":"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8","Type":"ContainerStarted","Data":"5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc"} Oct 30 00:48:53.142820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:53.142773 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" podStartSLOduration=2.142758283 podStartE2EDuration="2.142758283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:53.141483631 +0000 UTC m=+871.053610782" watchObservedRunningTime="2026-10-30 00:48:53.142758283 +0000 UTC m=+871.054885424" Oct 30 00:48:53.690038 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:53.690004 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:48:53.691794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:53.691769 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:48:53.691921 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:53.691821 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 00:48:54.651584 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:54.651550 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:54.690301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:54.690265 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:48:54.690575 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:54.690549 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 00:48:55.690150 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:55.690110 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:48:55.690559 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:55.690179 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 00:48:56.439620 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:56.439583 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:56.691285 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:56.691191 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:48:56.691913 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:56.691877 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 00:48:57.406167 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.406132 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2"] Oct 30 00:48:57.410996 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.410973 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.425379 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.425274 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2"] Oct 30 00:48:57.433136 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.433115 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-model-cache\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.433237 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.433152 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554qp\" (UniqueName: \"kubernetes.io/projected/645b8a00-6e13-426f-9474-a2e36a2b943e-kube-api-access-554qp\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.433237 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.433171 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-home\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.433237 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.433195 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-tmp-dir\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.433336 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.433244 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-dshm\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.433336 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.433295 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645b8a00-6e13-426f-9474-a2e36a2b943e-tls-certs\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.433868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.433852 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"distinct-w5-89a676-kserve-self-signed-certs\"" Oct 30 00:48:57.474682 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.463142 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:57.534582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.534549 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-tmp-dir\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.534934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.534911 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-dshm\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.535409 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.535058 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-tmp-dir\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.535580 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.535562 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645b8a00-6e13-426f-9474-a2e36a2b943e-tls-certs\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.535915 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.535898 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-model-cache\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.536073 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.536056 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554qp\" (UniqueName: \"kubernetes.io/projected/645b8a00-6e13-426f-9474-a2e36a2b943e-kube-api-access-554qp\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.536216 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.536200 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-home\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.536365 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.536256 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-model-cache\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.536514 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.536490 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-home\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.538224 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.538207 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-dshm\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.539349 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.539322 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645b8a00-6e13-426f-9474-a2e36a2b943e-tls-certs\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.561743 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.557831 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554qp\" (UniqueName: \"kubernetes.io/projected/645b8a00-6e13-426f-9474-a2e36a2b943e-kube-api-access-554qp\") pod \"distinct-w5-89a676-kserve-759485df45-4x6s2\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:57.691223 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.691127 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:48:57.691486 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.691456 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 00:48:57.726156 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:57.726120 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:48:58.065628 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:58.065593 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2"] Oct 30 00:48:58.069171 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:48:58.069143 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod645b8a00_6e13_426f_9474_a2e36a2b943e.slice/crio-7251b36d77c4247127c8832ddb9b54a4594ab1d5d2182337ee196152ac4b1af2 WatchSource:0}: Error finding container 7251b36d77c4247127c8832ddb9b54a4594ab1d5d2182337ee196152ac4b1af2: Status 404 returned error can't find the container with id 7251b36d77c4247127c8832ddb9b54a4594ab1d5d2182337ee196152ac4b1af2 Oct 30 00:48:58.120578 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:58.120537 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" event={"ID":"645b8a00-6e13-426f-9474-a2e36a2b943e","Type":"ContainerStarted","Data":"7251b36d77c4247127c8832ddb9b54a4594ab1d5d2182337ee196152ac4b1af2"} Oct 30 00:48:58.335391 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:58.335309 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:58.690269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:58.690166 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 00:48:58.690269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:58.690236 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 00:48:59.127169 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:59.127133 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" event={"ID":"645b8a00-6e13-426f-9474-a2e36a2b943e","Type":"ContainerStarted","Data":"639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70"} Oct 30 00:48:59.165210 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:59.165151 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podStartSLOduration=2.165132603 podStartE2EDuration="2.165132603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:59.163840659 +0000 UTC m=+877.075967828" watchObservedRunningTime="2026-10-30 00:48:59.165132603 +0000 UTC m=+877.077259745" Oct 30 00:48:59.726868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:48:59.726835 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:49:00.016720 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.016610 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:00.525915 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.525880 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:49:00.526795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.526769 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:49:00.527798 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.527777 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 00:49:00.585045 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.585017 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv"] Oct 30 00:49:00.589264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.589232 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.606678 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.602400 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"distinct-w3-b3593a-kserve-self-signed-certs\"" Oct 30 00:49:00.606678 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.605085 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv"] Oct 30 00:49:00.659731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.659431 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:00.676383 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.676345 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-dshm\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.676555 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.676386 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67q8b\" (UniqueName: \"kubernetes.io/projected/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-kube-api-access-67q8b\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.676555 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.676419 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tls-certs\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.676555 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.676475 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-model-cache\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.676555 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.676507 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tmp-dir\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.676555 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.676542 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-home\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.778149 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.778042 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tmp-dir\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.778149 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.778138 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-home\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.778394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.778222 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-dshm\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.778394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.778251 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67q8b\" (UniqueName: \"kubernetes.io/projected/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-kube-api-access-67q8b\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.778394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.778282 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tls-certs\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.778394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.778375 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-model-cache\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.779325 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:49:00.778847 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:49:00.779325 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.778871 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-model-cache\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.780208 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.779909 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tmp-dir\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.780208 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.780167 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-home\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.783342 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.783299 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-dshm\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.784676 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.784528 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tls-certs\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.803224 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.803173 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67q8b\" (UniqueName: \"kubernetes.io/projected/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-kube-api-access-67q8b\") pod \"distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:00.910198 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:00.909935 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:01.279617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:01.279542 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv"] Oct 30 00:49:01.282871 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:49:01.282845 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89ae0bf1_3213_46b0_8d0c_58ce5fb889a9.slice/crio-bbaac542aa84471f3553825f630274d4df32431e683287a1cc4bc30201561c71 WatchSource:0}: Error finding container bbaac542aa84471f3553825f630274d4df32431e683287a1cc4bc30201561c71: Status 404 returned error can't find the container with id bbaac542aa84471f3553825f630274d4df32431e683287a1cc4bc30201561c71 Oct 30 00:49:01.909824 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:01.909748 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:02.143880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:02.143840 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" event={"ID":"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9","Type":"ContainerStarted","Data":"ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb"} Oct 30 00:49:02.143880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:02.143883 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" event={"ID":"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9","Type":"ContainerStarted","Data":"bbaac542aa84471f3553825f630274d4df32431e683287a1cc4bc30201561c71"} Oct 30 00:49:02.183231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:02.183121 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podStartSLOduration=2.183104982 podStartE2EDuration="2.183104982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:02.181350344 +0000 UTC m=+880.093477484" watchObservedRunningTime="2026-10-30 00:49:02.183104982 +0000 UTC m=+880.095232122" Oct 30 00:49:02.910901 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:02.910868 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:03.395036 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.394999 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7"] Oct 30 00:49:03.398560 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.398541 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.426113 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.426079 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7"] Oct 30 00:49:03.509210 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.509176 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.509403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.509220 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8fd775-d621-44cd-84b2-9030019d7f44-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.509403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.509333 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.509403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.509371 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.509523 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.509437 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.509523 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.509506 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lgz\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-kube-api-access-r9lgz\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.509598 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.509543 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.509598 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.509565 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.509674 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.509618 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.551508 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.551473 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:03.611226 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611182 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8fd775-d621-44cd-84b2-9030019d7f44-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611266 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611297 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611343 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611374 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lgz\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-kube-api-access-r9lgz\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611418 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611443 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611491 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611574 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611839 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.611988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.611947 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.612139 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.612069 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.612286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.612263 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8fd775-d621-44cd-84b2-9030019d7f44-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.612696 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.612670 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.614737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.614680 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.614842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.614769 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.635377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.635345 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.635662 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.635641 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lgz\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-kube-api-access-r9lgz\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.655419 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.655337 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22"] Oct 30 00:49:03.660012 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.659991 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.687013 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.686852 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22"] Oct 30 00:49:03.712437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.712405 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.712613 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.712449 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.712613 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.712590 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.712817 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.712629 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b387e5c-219f-4249-a3ee-4903af9a493f-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.712817 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.712730 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb44q\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-kube-api-access-xb44q\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.712817 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.712776 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.712948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.712864 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.712948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.712901 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.713032 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.712990 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.714236 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.714214 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:03.814615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.814581 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.814801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.814628 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.814801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.814698 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.814801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.814766 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b387e5c-219f-4249-a3ee-4903af9a493f-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.814801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.814795 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb44q\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-kube-api-access-xb44q\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.815057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.814848 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.815057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.814895 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.815057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.814932 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.815057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.815023 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.815057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.815045 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.815860 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.815356 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.815860 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.815826 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.816091 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.816056 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.816161 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.816135 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b387e5c-219f-4249-a3ee-4903af9a493f-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.817311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.817284 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.818352 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.818332 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.836202 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.836182 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.836380 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.836363 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb44q\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-kube-api-access-xb44q\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:03.977324 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:03.977253 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:04.035626 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.035596 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7"] Oct 30 00:49:04.048311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.048270 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:49:04.048426 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.048364 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:49:04.048426 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.048409 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:49:04.156665 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.156620 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" event={"ID":"1c8fd775-d621-44cd-84b2-9030019d7f44","Type":"ContainerStarted","Data":"ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21"} Oct 30 00:49:04.156817 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.156690 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" event={"ID":"1c8fd775-d621-44cd-84b2-9030019d7f44","Type":"ContainerStarted","Data":"e5285bf1c930872d725472d9301e9661425ff4d5c5cb9f5c2f0407d6de05c39c"} Oct 30 00:49:04.200162 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.200108 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" podStartSLOduration=1.2000927479999999 podStartE2EDuration="1.200092748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:04.196637678 +0000 UTC m=+882.108764819" watchObservedRunningTime="2026-10-30 00:49:04.200092748 +0000 UTC m=+882.112219889" Oct 30 00:49:04.286839 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.285329 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22"] Oct 30 00:49:04.292265 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:49:04.292235 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b387e5c_219f_4249_a3ee_4903af9a493f.slice/crio-a4f572a6b2dcef739568a6f8431c0155292195199f16d5c168515d5e7bb876d7 WatchSource:0}: Error finding container a4f572a6b2dcef739568a6f8431c0155292195199f16d5c168515d5e7bb876d7: Status 404 returned error can't find the container with id a4f572a6b2dcef739568a6f8431c0155292195199f16d5c168515d5e7bb876d7 Oct 30 00:49:04.295482 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.295435 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:49:04.295586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.295527 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:49:04.295891 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.295866 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:49:04.714570 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:04.714485 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:05.219648 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:05.219461 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" event={"ID":"0b387e5c-219f-4249-a3ee-4903af9a493f","Type":"ContainerStarted","Data":"adcee1d7d371cc3f3d8183e9463d484463d3ca36ff67ea5f229d0e4c1679c447"} Oct 30 00:49:05.219648 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:05.219511 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" event={"ID":"0b387e5c-219f-4249-a3ee-4903af9a493f","Type":"ContainerStarted","Data":"a4f572a6b2dcef739568a6f8431c0155292195199f16d5c168515d5e7bb876d7"} Oct 30 00:49:05.282733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:05.282012 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" podStartSLOduration=2.281992788 podStartE2EDuration="2.281992788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:05.279245499 +0000 UTC m=+883.191372641" watchObservedRunningTime="2026-10-30 00:49:05.281992788 +0000 UTC m=+883.194119931" Oct 30 00:49:05.531393 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:05.531277 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:05.719733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:05.719659 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:49:05.719913 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:05.719747 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 00:49:05.982732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:05.981697 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:05.982732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:05.981887 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:49:05.982732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:05.981937 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 00:49:06.716057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:06.716022 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:49:06.716667 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:06.716631 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 00:49:06.980800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:06.980157 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:49:06.980800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:06.980234 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 00:49:07.246080 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:07.245999 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:07.716735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:07.715130 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:49:07.716735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:07.715192 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 00:49:07.726941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:07.726905 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:49:07.739702 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:07.739675 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:49:07.979790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:07.978941 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:49:07.979790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:07.979025 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 00:49:08.253766 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:08.253688 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:49:08.716130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:08.716093 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:49:08.716496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:08.716457 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 00:49:08.979322 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:08.979216 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:49:08.979977 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:08.979937 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 00:49:09.365647 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:09.365608 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:09.715790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:09.715114 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:49:09.715790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:09.715175 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 00:49:09.980795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:09.980407 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:49:09.980795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:09.980467 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 00:49:10.714758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:10.714696 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:49:10.714992 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:10.714797 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 00:49:10.911071 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:10.911031 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:10.929045 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:10.929014 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:11.236269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:11.236238 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:11.236683 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:11.236566 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:11.237331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:11.237314 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 00:49:11.239205 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:11.239174 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:11.270481 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:11.270459 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:49:11.721543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:11.721509 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:11.723177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:11.723015 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:11.724177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:11.724125 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 00:49:13.329940 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:13.329902 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.627386 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:14.627342 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.904691 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:14.904611 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:16.582388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:16.582354 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:18.411739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:18.411677 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:19.356972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:19.356941 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:20.127190 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:20.127159 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:21.827213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:21.827166 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:23.453231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.453169 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:23.610455 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.610423 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5"] Oct 30 00:49:23.615633 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.615605 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.616211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.616187 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ace35f-pb2h5\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.616295 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.616235 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5sld\" (UniqueName: \"kubernetes.io/projected/e83e291a-a8de-44e2-91a9-3cd4824f6a97-kube-api-access-k5sld\") pod \"maas-api-revoke-sub-a7ace35f-pb2h5\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.616336 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.616293 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ace35f-pb2h5\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.643867 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.643841 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:49:23.644013 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.643876 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:49:23.646498 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.646473 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5"] Oct 30 00:49:23.717588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.717517 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ace35f-pb2h5\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.717588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.717563 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5sld\" (UniqueName: \"kubernetes.io/projected/e83e291a-a8de-44e2-91a9-3cd4824f6a97-kube-api-access-k5sld\") pod \"maas-api-revoke-sub-a7ace35f-pb2h5\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.717588 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.717582 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ace35f-pb2h5\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.718240 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.718212 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ace35f-pb2h5\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.720401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.720376 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ace35f-pb2h5\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.739348 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.739322 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5sld\" (UniqueName: \"kubernetes.io/projected/e83e291a-a8de-44e2-91a9-3cd4824f6a97-kube-api-access-k5sld\") pod \"maas-api-revoke-sub-a7ace35f-pb2h5\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:23.928681 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:23.928646 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:24.235038 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:49:24.235000 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83e291a_a8de_44e2_91a9_3cd4824f6a97.slice/crio-631a6a7cd11665f31143b602f736b77974552996094e5c7fa0766fb5f70fe6f2 WatchSource:0}: Error finding container 631a6a7cd11665f31143b602f736b77974552996094e5c7fa0766fb5f70fe6f2: Status 404 returned error can't find the container with id 631a6a7cd11665f31143b602f736b77974552996094e5c7fa0766fb5f70fe6f2 Oct 30 00:49:24.235811 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:24.235789 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5"] Oct 30 00:49:24.348369 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:24.348333 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" event={"ID":"e83e291a-a8de-44e2-91a9-3cd4824f6a97","Type":"ContainerStarted","Data":"631a6a7cd11665f31143b602f736b77974552996094e5c7fa0766fb5f70fe6f2"} Oct 30 00:49:25.138567 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:25.138540 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:25.356752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:25.356641 4517 generic.go:356] "Generic (PLEG): container finished" podID="e83e291a-a8de-44e2-91a9-3cd4824f6a97" containerID="a8ae7f526f01762f0f4dd0fc15636151e0aa712768daca8964ae87abbd639b0e" exitCode=0 Oct 30 00:49:25.356752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:25.356692 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" event={"ID":"e83e291a-a8de-44e2-91a9-3cd4824f6a97","Type":"ContainerDied","Data":"a8ae7f526f01762f0f4dd0fc15636151e0aa712768daca8964ae87abbd639b0e"} Oct 30 00:49:26.517525 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.517503 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:26.548039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.548011 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-xks-ca\") pod \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " Oct 30 00:49:26.548186 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.548052 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83e291a-a8de-44e2-91a9-3cd4824f6a97-kube-api-access-k5sld\" (UniqueName: \"kubernetes.io/projected/e83e291a-a8de-44e2-91a9-3cd4824f6a97-kube-api-access-k5sld\") pod \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " Oct 30 00:49:26.548186 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.548166 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-openshift-ca\") pod \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\" (UID: \"e83e291a-a8de-44e2-91a9-3cd4824f6a97\") " Oct 30 00:49:26.548497 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.548473 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-openshift-ca" pod "e83e291a-a8de-44e2-91a9-3cd4824f6a97" (UID: "e83e291a-a8de-44e2-91a9-3cd4824f6a97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:49:26.548634 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.548615 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:49:26.550309 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.550287 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-xks-ca" pod "e83e291a-a8de-44e2-91a9-3cd4824f6a97" (UID: "e83e291a-a8de-44e2-91a9-3cd4824f6a97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:49:26.550400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.550355 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83e291a-a8de-44e2-91a9-3cd4824f6a97-kube-api-access-k5sld" pod "e83e291a-a8de-44e2-91a9-3cd4824f6a97" (UID: "e83e291a-a8de-44e2-91a9-3cd4824f6a97"). InnerVolumeSpecName "kube-api-access-k5sld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:49:26.649277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.649204 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e291a-a8de-44e2-91a9-3cd4824f6a97-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:49:26.649277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.649233 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5sld\" (UniqueName: \"kubernetes.io/projected/e83e291a-a8de-44e2-91a9-3cd4824f6a97-kube-api-access-k5sld\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:49:26.779349 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:26.779309 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:27.369563 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:27.369533 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" Oct 30 00:49:27.369779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:27.369534 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5" event={"ID":"e83e291a-a8de-44e2-91a9-3cd4824f6a97","Type":"ContainerDied","Data":"631a6a7cd11665f31143b602f736b77974552996094e5c7fa0766fb5f70fe6f2"} Oct 30 00:49:27.369779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:27.369673 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="631a6a7cd11665f31143b602f736b77974552996094e5c7fa0766fb5f70fe6f2" Oct 30 00:49:28.523255 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:28.523215 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:30.328010 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:30.327972 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:30.758951 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:49:30.758857 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:49:31.721752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:31.718903 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:32.145676 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:32.145644 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:33.731887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:33.731849 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:33.964742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:33.964684 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq"] Oct 30 00:49:33.968330 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:33.965969 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e83e291a-a8de-44e2-91a9-3cd4824f6a97" containerName="revoke-keys" Oct 30 00:49:33.968330 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:33.965993 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83e291a-a8de-44e2-91a9-3cd4824f6a97" containerName="revoke-keys" Oct 30 00:49:33.968330 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:33.966332 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83e291a-a8de-44e2-91a9-3cd4824f6a97" containerName="revoke-keys" Oct 30 00:49:33.976106 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:33.976073 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:33.994343 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:33.994320 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq"] Oct 30 00:49:34.001423 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.001394 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-89a676-openshift-default-dockercfg-2l6sv\"" Oct 30 00:49:34.039489 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.039450 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.039652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.039561 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.039652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.039591 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrlq\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-kube-api-access-cxrlq\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.039809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.039656 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.039809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.039784 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.039957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.039834 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.039957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.039863 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.039957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.039932 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.039957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.039951 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47e978-79d1-4d55-a14a-31ec5c412222-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.141488 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.141460 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.141677 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.141514 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.141677 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.141539 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrlq\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-kube-api-access-cxrlq\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.141677 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.141570 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.141677 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.141617 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.141934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.141827 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.141934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.141870 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.142041 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.141991 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.142041 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.141995 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.142160 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.142047 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47e978-79d1-4d55-a14a-31ec5c412222-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.142363 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.142338 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.142534 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.142366 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.142534 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.142341 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.142912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.142895 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47e978-79d1-4d55-a14a-31ec5c412222-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.144100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.144081 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.144537 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.144511 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.167503 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.167469 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.167608 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.167551 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrlq\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-kube-api-access-cxrlq\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.291446 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.291416 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:34.614676 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.614588 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq"] Oct 30 00:49:34.619276 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:49:34.619239 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d47e978_79d1_4d55_a14a_31ec5c412222.slice/crio-2d1d96636493845f4c31c7094d76ada617f5910d9b622e6ba524d5bd687af56f WatchSource:0}: Error finding container 2d1d96636493845f4c31c7094d76ada617f5910d9b622e6ba524d5bd687af56f: Status 404 returned error can't find the container with id 2d1d96636493845f4c31c7094d76ada617f5910d9b622e6ba524d5bd687af56f Oct 30 00:49:34.621636 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.621604 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:49:34.621758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.621668 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:49:34.621758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:34.621697 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:49:35.433453 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:35.433417 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" event={"ID":"2d47e978-79d1-4d55-a14a-31ec5c412222","Type":"ContainerStarted","Data":"c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b"} Oct 30 00:49:35.433952 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:35.433475 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" event={"ID":"2d47e978-79d1-4d55-a14a-31ec5c412222","Type":"ContainerStarted","Data":"2d1d96636493845f4c31c7094d76ada617f5910d9b622e6ba524d5bd687af56f"} Oct 30 00:49:35.497230 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:35.497160 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" podStartSLOduration=2.497145916 podStartE2EDuration="2.497145916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:35.495517557 +0000 UTC m=+913.407644710" watchObservedRunningTime="2026-10-30 00:49:35.497145916 +0000 UTC m=+913.409273067" Oct 30 00:49:35.594540 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:35.594495 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:36.002015 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.001970 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t"] Oct 30 00:49:36.005782 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.005761 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.018734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.018695 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:49:36.029239 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.029218 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:49:36.033212 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.033188 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t"] Oct 30 00:49:36.093625 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.093300 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2lm\" (UniqueName: \"kubernetes.io/projected/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-kube-api-access-4r2lm\") pod \"maas-api-revoke-sub-ebcc7d03-jkp5t\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.093625 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.093345 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebcc7d03-jkp5t\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.093625 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.093468 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebcc7d03-jkp5t\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.194653 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.194608 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebcc7d03-jkp5t\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.194884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.194752 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2lm\" (UniqueName: \"kubernetes.io/projected/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-kube-api-access-4r2lm\") pod \"maas-api-revoke-sub-ebcc7d03-jkp5t\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.194884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.194792 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebcc7d03-jkp5t\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.195597 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.195569 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebcc7d03-jkp5t\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.197334 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.197302 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebcc7d03-jkp5t\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.217405 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.217374 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2lm\" (UniqueName: \"kubernetes.io/projected/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-kube-api-access-4r2lm\") pod \"maas-api-revoke-sub-ebcc7d03-jkp5t\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.292329 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.292295 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:36.302742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.293609 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:49:36.302742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.293664 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 00:49:36.319566 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.319543 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:36.655476 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:36.655407 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t"] Oct 30 00:49:37.291744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:37.286206 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:37.296228 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:37.295775 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:49:37.296228 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:37.295833 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 00:49:37.448204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:37.448165 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" event={"ID":"0c62e9b9-b4f1-4560-8179-26dbfa1120d5","Type":"ContainerStarted","Data":"665c2d4c2c1d3fe390b1c63b382070125e45d41eedcff0f2240ccca654d15e7f"} Oct 30 00:49:37.448408 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:37.448247 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" event={"ID":"0c62e9b9-b4f1-4560-8179-26dbfa1120d5","Type":"ContainerStarted","Data":"edf9bf525e6c2a07d5eb9ee76d2da35d516e5b74e6fe68d7d6a764bbbe800ddf"} Oct 30 00:49:37.496727 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:37.496653 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" podStartSLOduration=2.496638698 podStartE2EDuration="2.496638698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:37.495302889 +0000 UTC m=+915.407430030" watchObservedRunningTime="2026-10-30 00:49:37.496638698 +0000 UTC m=+915.408765839" Oct 30 00:49:38.294054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:38.294013 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:49:38.294575 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:38.294085 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 00:49:38.455453 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:38.455414 4517 generic.go:356] "Generic (PLEG): container finished" podID="0c62e9b9-b4f1-4560-8179-26dbfa1120d5" containerID="665c2d4c2c1d3fe390b1c63b382070125e45d41eedcff0f2240ccca654d15e7f" exitCode=0 Oct 30 00:49:38.455615 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:38.455460 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" event={"ID":"0c62e9b9-b4f1-4560-8179-26dbfa1120d5","Type":"ContainerDied","Data":"665c2d4c2c1d3fe390b1c63b382070125e45d41eedcff0f2240ccca654d15e7f"} Oct 30 00:49:39.077740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.076135 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:39.292763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.292288 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:49:39.292763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.292343 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 00:49:39.691875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.691838 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:39.740983 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.740903 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-kube-api-access-4r2lm\" (UniqueName: \"kubernetes.io/projected/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-kube-api-access-4r2lm\") pod \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " Oct 30 00:49:39.740983 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.740948 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-openshift-ca\") pod \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " Oct 30 00:49:39.741217 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.741117 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-xks-ca\") pod \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\" (UID: \"0c62e9b9-b4f1-4560-8179-26dbfa1120d5\") " Oct 30 00:49:39.741384 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.741362 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-openshift-ca" pod "0c62e9b9-b4f1-4560-8179-26dbfa1120d5" (UID: "0c62e9b9-b4f1-4560-8179-26dbfa1120d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:49:39.741549 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.741533 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:49:39.743479 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.743448 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-xks-ca" pod "0c62e9b9-b4f1-4560-8179-26dbfa1120d5" (UID: "0c62e9b9-b4f1-4560-8179-26dbfa1120d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:49:39.743820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.743795 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-kube-api-access-4r2lm" pod "0c62e9b9-b4f1-4560-8179-26dbfa1120d5" (UID: "0c62e9b9-b4f1-4560-8179-26dbfa1120d5"). InnerVolumeSpecName "kube-api-access-4r2lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:49:39.842187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.842155 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:49:39.842187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:39.842182 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r2lm\" (UniqueName: \"kubernetes.io/projected/0c62e9b9-b4f1-4560-8179-26dbfa1120d5-kube-api-access-4r2lm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:49:40.316775 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:40.315847 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:49:40.316775 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:40.315968 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 00:49:40.492254 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:40.492213 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" event={"ID":"0c62e9b9-b4f1-4560-8179-26dbfa1120d5","Type":"ContainerDied","Data":"edf9bf525e6c2a07d5eb9ee76d2da35d516e5b74e6fe68d7d6a764bbbe800ddf"} Oct 30 00:49:40.492437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:40.492288 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf9bf525e6c2a07d5eb9ee76d2da35d516e5b74e6fe68d7d6a764bbbe800ddf" Oct 30 00:49:40.492437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:40.492340 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t" Oct 30 00:49:40.874328 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:40.874284 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:41.294019 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:41.293981 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 00:49:41.294303 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:41.294273 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 00:49:42.681778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:42.681733 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:43.334853 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:43.334822 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:49:43.335063 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:43.334872 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:43.640664 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:43.640571 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:43.641090 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:43.641070 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:43.642278 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:43.642258 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 00:49:43.985411 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:43.985321 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:44.410868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:44.410816 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:46.249175 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:46.249144 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:47.913996 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:47.913966 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:48.438554 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.438520 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx"] Oct 30 00:49:48.439073 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.439046 4517 cpu_manager.go:414] "containerMap: removing container" podUID="0c62e9b9-b4f1-4560-8179-26dbfa1120d5" containerName="revoke-keys" Oct 30 00:49:48.439179 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.439076 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c62e9b9-b4f1-4560-8179-26dbfa1120d5" containerName="revoke-keys" Oct 30 00:49:48.439242 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.439185 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c62e9b9-b4f1-4560-8179-26dbfa1120d5" containerName="revoke-keys" Oct 30 00:49:48.442754 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.442734 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.466786 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.466767 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"distinct-2-w3-b3593a-kserve-self-signed-certs\"" Oct 30 00:49:48.470668 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.470645 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx"] Oct 30 00:49:48.491297 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.491264 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:48.567907 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.567874 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-dshm\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.568109 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.567911 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtnzd\" (UniqueName: \"kubernetes.io/projected/9c3e0328-65f0-4890-b71d-242bea9998c8-kube-api-access-mtnzd\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.568109 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.568089 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-model-cache\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.568278 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.568254 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3e0328-65f0-4890-b71d-242bea9998c8-tls-certs\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.568340 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.568308 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-home\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.568397 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.568381 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-tmp-dir\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.669372 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.669341 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-tmp-dir\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.669726 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.669686 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-dshm\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.669879 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.669848 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtnzd\" (UniqueName: \"kubernetes.io/projected/9c3e0328-65f0-4890-b71d-242bea9998c8-kube-api-access-mtnzd\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.669956 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.669903 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-model-cache\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.670042 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.670025 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3e0328-65f0-4890-b71d-242bea9998c8-tls-certs\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.670119 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.670070 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-home\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.671470 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.669781 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-tmp-dir\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.671470 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.670431 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-home\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.671470 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.670726 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-model-cache\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.686270 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.686243 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3e0328-65f0-4890-b71d-242bea9998c8-tls-certs\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.693610 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.693538 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtnzd\" (UniqueName: \"kubernetes.io/projected/9c3e0328-65f0-4890-b71d-242bea9998c8-kube-api-access-mtnzd\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.697779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.697751 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-dshm\") pod \"distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:48.756971 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:48.756942 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:49.075343 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:49.075312 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx"] Oct 30 00:49:49.082749 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:49:49.082688 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c3e0328_65f0_4890_b71d_242bea9998c8.slice/crio-7e9f7f9506dc064f024571696558dee07f15feb7a7efce095a77bb7f245d48a7 WatchSource:0}: Error finding container 7e9f7f9506dc064f024571696558dee07f15feb7a7efce095a77bb7f245d48a7: Status 404 returned error can't find the container with id 7e9f7f9506dc064f024571696558dee07f15feb7a7efce095a77bb7f245d48a7 Oct 30 00:49:49.552941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:49.552898 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" event={"ID":"9c3e0328-65f0-4890-b71d-242bea9998c8","Type":"ContainerStarted","Data":"73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7"} Oct 30 00:49:49.553132 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:49.552950 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" event={"ID":"9c3e0328-65f0-4890-b71d-242bea9998c8","Type":"ContainerStarted","Data":"7e9f7f9506dc064f024571696558dee07f15feb7a7efce095a77bb7f245d48a7"} Oct 30 00:49:49.590756 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:49.590691 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podStartSLOduration=1.590677922 podStartE2EDuration="1.590677922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:49.589810249 +0000 UTC m=+927.501937392" watchObservedRunningTime="2026-10-30 00:49:49.590677922 +0000 UTC m=+927.502805092" Oct 30 00:49:49.716552 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:49.716521 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:49.757609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:49.757435 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:51.381166 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:51.381119 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:52.958291 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:52.958248 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:54.587423 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:54.587386 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:56.323084 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:56.323044 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:58.005100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:58.005066 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:58.759027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:58.758141 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:58.925673 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:58.925643 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:49:59.646853 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:49:59.646826 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:50:00.144080 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:00.144039 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:00.758802 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:50:00.758768 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:50:01.795456 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:01.795413 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:03.449551 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.449520 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:03.706944 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.706859 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv"] Oct 30 00:50:03.711141 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.711118 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.740753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.740692 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv"] Oct 30 00:50:03.776898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.776864 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb760a4-2540-4d84-b7d0-526f0b08985e-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.777138 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.777119 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59r6\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-kube-api-access-f59r6\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.777368 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.777355 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.777460 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.777448 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.777551 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.777539 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.777651 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.777639 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.777797 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.777783 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.777918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.777906 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.778010 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.777998 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.879345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.879302 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.879524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.879358 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.879524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.879408 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb760a4-2540-4d84-b7d0-526f0b08985e-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.879524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.879444 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59r6\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-kube-api-access-f59r6\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.879690 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.879576 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.879690 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.879599 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.879690 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.879624 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.879690 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.879654 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.879958 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.879690 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.880331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.880304 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.880456 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.880325 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb760a4-2540-4d84-b7d0-526f0b08985e-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.880583 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.880561 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.880688 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.880664 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.880827 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.880809 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.882462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.882435 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.883164 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.883142 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.901122 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.901086 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:03.901717 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:03.901681 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59r6\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-kube-api-access-f59r6\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:04.029061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.028979 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:04.353530 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.353504 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv"] Oct 30 00:50:04.355328 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:50:04.355264 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb760a4_2540_4d84_b7d0_526f0b08985e.slice/crio-3647bbe414d5d546c8623cd200957dc807b4fff2ed366e783522b9ef0b4d5b83 WatchSource:0}: Error finding container 3647bbe414d5d546c8623cd200957dc807b4fff2ed366e783522b9ef0b4d5b83: Status 404 returned error can't find the container with id 3647bbe414d5d546c8623cd200957dc807b4fff2ed366e783522b9ef0b4d5b83 Oct 30 00:50:04.357918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.357878 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:04.358029 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.357951 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:04.358029 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.357982 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:04.664265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.664183 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" event={"ID":"adb760a4-2540-4d84-b7d0-526f0b08985e","Type":"ContainerStarted","Data":"d379c988c4115ebcf5531289c757b536d0b40e06e0c1a58264bad7cd4b6ab8b4"} Oct 30 00:50:04.664265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.664236 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" event={"ID":"adb760a4-2540-4d84-b7d0-526f0b08985e","Type":"ContainerStarted","Data":"3647bbe414d5d546c8623cd200957dc807b4fff2ed366e783522b9ef0b4d5b83"} Oct 30 00:50:04.696747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.696688 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn"] Oct 30 00:50:04.701210 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.701185 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.721370 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.721336 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn"] Oct 30 00:50:04.729730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.728362 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"distinct-w1-beb8d5-kserve-self-signed-certs\"" Oct 30 00:50:04.757680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.757645 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:04.770238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.770186 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podStartSLOduration=1.770169133 podStartE2EDuration="1.770169133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:04.733084796 +0000 UTC m=+942.645211938" watchObservedRunningTime="2026-10-30 00:50:04.770169133 +0000 UTC m=+942.682296274" Oct 30 00:50:04.791308 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.791262 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tls-certs\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.791541 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.791337 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-home\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.791541 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.791441 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-model-cache\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.791541 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.791482 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-dshm\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.791541 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.791520 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4nfl\" (UniqueName: \"kubernetes.io/projected/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-kube-api-access-t4nfl\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.791799 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.791615 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tmp-dir\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.898792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.894390 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tmp-dir\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.898792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.894503 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tls-certs\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.898792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.894565 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-home\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.898792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.894604 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-model-cache\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.898792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.894644 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-dshm\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.898792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.894684 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nfl\" (UniqueName: \"kubernetes.io/projected/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-kube-api-access-t4nfl\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.898792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.895480 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tmp-dir\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.898792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.896146 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-model-cache\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.898792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.896379 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-home\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.917991 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.905094 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tls-certs\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.917991 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.917851 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-dshm\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:04.920435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:04.918569 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4nfl\" (UniqueName: \"kubernetes.io/projected/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-kube-api-access-t4nfl\") pod \"distinct-w1-beb8d5-kserve-5476dc9844-mr4sn\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:05.021832 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:05.017014 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:05.032778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:05.029560 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:05.032778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:05.031319 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:50:05.032778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:05.031374 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 00:50:05.330215 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:05.330184 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:05.400845 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:05.400782 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn"] Oct 30 00:50:05.404550 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:50:05.404510 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbacc172f_6097_4a9d_bf9e_cbc32a88bf4e.slice/crio-2da28cc53a2adf0b7773362a03be7ff69f5e688df187656d54b380bc31b5b046 WatchSource:0}: Error finding container 2da28cc53a2adf0b7773362a03be7ff69f5e688df187656d54b380bc31b5b046: Status 404 returned error can't find the container with id 2da28cc53a2adf0b7773362a03be7ff69f5e688df187656d54b380bc31b5b046 Oct 30 00:50:05.688608 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:05.688508 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" event={"ID":"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e","Type":"ContainerStarted","Data":"b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618"} Oct 30 00:50:05.688608 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:05.688568 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" event={"ID":"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e","Type":"ContainerStarted","Data":"2da28cc53a2adf0b7773362a03be7ff69f5e688df187656d54b380bc31b5b046"} Oct 30 00:50:05.727809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:05.727761 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podStartSLOduration=1.727747251 podStartE2EDuration="1.727747251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:05.726639795 +0000 UTC m=+943.638766936" watchObservedRunningTime="2026-10-30 00:50:05.727747251 +0000 UTC m=+943.639874391" Oct 30 00:50:06.022293 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:06.022211 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:06.029622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:06.029594 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:50:06.029751 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:06.029641 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 00:50:07.029668 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:07.029628 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:50:07.030191 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:07.029739 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 00:50:07.643277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:07.643237 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:08.029668 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:08.029575 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:50:08.029668 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:08.029636 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 00:50:09.029291 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:09.029257 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:50:09.029472 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:09.029308 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 00:50:10.029526 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:10.029492 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:50:10.029974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:10.029548 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 00:50:10.039798 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:10.039762 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:11.029895 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:11.029855 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:50:11.030277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:11.029920 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 00:50:11.836378 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:11.836337 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:12.034143 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:12.034110 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:12.034562 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:12.034436 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:12.035026 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:12.035007 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 00:50:13.515651 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:13.515619 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:13.787528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:13.787475 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:15.022645 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:15.022558 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:15.035264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:15.035237 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:15.232914 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:15.232881 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:15.756501 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:15.756473 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:16.914080 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:16.914042 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:18.262011 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.261930 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4"] Oct 30 00:50:18.268567 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.268532 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.293929 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.293899 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-40ac41-openshift-default-dockercfg-kr4bx\"" Oct 30 00:50:18.297328 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.297301 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4"] Oct 30 00:50:18.359147 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.359114 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.359349 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.359274 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.359349 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.359338 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68d521f-b88b-40e6-8f15-f822d8394f55-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.359463 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.359377 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.359463 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.359438 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.359566 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.359474 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.359683 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.359665 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.359761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.359698 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.359761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.359732 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwgk\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-kube-api-access-fmwgk\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.374196 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.374164 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz"] Oct 30 00:50:18.378083 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.378062 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.391183 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.391153 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5062f6-openshift-default-dockercfg-9bf6d\"" Oct 30 00:50:18.402867 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.402838 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz"] Oct 30 00:50:18.460790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.460752 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.460790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.460792 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.460815 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.461061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.460859 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvr2k\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-kube-api-access-nvr2k\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.461061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.460894 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.461061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461002 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dadf0c-1fee-4fec-9080-c5816e329cde-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.461061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461022 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461036 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461057 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwgk\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-kube-api-access-fmwgk\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461094 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461141 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.461452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461176 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461228 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461239 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.461452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461346 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.461452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461360 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461387 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461459 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68d521f-b88b-40e6-8f15-f822d8394f55-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461508 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.461911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461533 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.461911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461586 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.461911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.461823 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.462179 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.462111 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68d521f-b88b-40e6-8f15-f822d8394f55-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.464298 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.464274 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.464631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.464608 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.483070 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.483039 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.483210 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.483146 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwgk\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-kube-api-access-fmwgk\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.562692 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.562649 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dadf0c-1fee-4fec-9080-c5816e329cde-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.562923 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.562722 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.562923 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.562771 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.562923 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.562897 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563009 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563048 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563115 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563181 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvr2k\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-kube-api-access-nvr2k\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563222 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563387 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563241 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563387 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563297 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563563 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563537 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dadf0c-1fee-4fec-9080-c5816e329cde-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563636 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563563 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.563696 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.563662 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.565637 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.565617 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.565827 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.565809 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.584954 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.584926 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.585080 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.584977 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvr2k\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-kube-api-access-nvr2k\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.585782 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.585756 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:18.690893 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.690828 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:18.837883 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.837807 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:18.856403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.856376 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp"] Oct 30 00:50:18.861475 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.861451 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:18.889286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.889245 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp"] Oct 30 00:50:18.913262 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.913232 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4"] Oct 30 00:50:18.916578 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:50:18.916545 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68d521f_b88b_40e6_8f15_f822d8394f55.slice/crio-df85ff1f67fda9c3a4f10965c6e5f545cda17234015df531f7e58ec774d1bdd6 WatchSource:0}: Error finding container df85ff1f67fda9c3a4f10965c6e5f545cda17234015df531f7e58ec774d1bdd6: Status 404 returned error can't find the container with id df85ff1f67fda9c3a4f10965c6e5f545cda17234015df531f7e58ec774d1bdd6 Oct 30 00:50:18.919174 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.919144 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:18.919261 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.919207 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:18.919261 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.919238 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:18.977180 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.977148 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:18.977359 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.977223 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkv6l\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-kube-api-access-qkv6l\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:18.977359 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.977333 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:18.977498 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.977388 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:18.977498 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.977450 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:18.977498 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.977475 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:18.978875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.977510 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:18.978875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.977559 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:18.978875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:18.977585 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.011264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.011226 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz"] Oct 30 00:50:19.014082 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:50:19.014046 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39dadf0c_1fee_4fec_9080_c5816e329cde.slice/crio-6b0e4ac015fe44b240c448fa9249a8d54f2e28803434a6239a009afe63a2de88 WatchSource:0}: Error finding container 6b0e4ac015fe44b240c448fa9249a8d54f2e28803434a6239a009afe63a2de88: Status 404 returned error can't find the container with id 6b0e4ac015fe44b240c448fa9249a8d54f2e28803434a6239a009afe63a2de88 Oct 30 00:50:19.016777 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.016743 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:19.016864 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.016812 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:19.016864 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.016845 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:19.079141 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079112 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.079265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079154 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.079265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079193 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.079265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079236 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.079435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079268 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.079435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079304 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.079435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079341 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkv6l\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-kube-api-access-qkv6l\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.079435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079364 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.079435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079405 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.079693 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.079670 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.080406 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.080378 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.080738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.080691 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.080801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.080698 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.080855 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.080792 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.082187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.082165 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.082534 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.082517 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.101783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.101686 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkv6l\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-kube-api-access-qkv6l\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.101783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.101697 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.186845 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.186808 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:19.596686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.595836 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp"] Oct 30 00:50:19.613212 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.613152 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:19.613349 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.613276 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:19.613417 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.613349 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:19.857791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.857184 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" event={"ID":"701f808d-ca6b-4e78-aa66-2516a24f0ed7","Type":"ContainerStarted","Data":"b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f"} Oct 30 00:50:19.857791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.857265 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" event={"ID":"701f808d-ca6b-4e78-aa66-2516a24f0ed7","Type":"ContainerStarted","Data":"9f9d282313e17da6f3ff8fe4a991dfd6993093ee85e7feb8cbaf9caee25dc4a9"} Oct 30 00:50:19.868680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.868637 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" event={"ID":"39dadf0c-1fee-4fec-9080-c5816e329cde","Type":"ContainerStarted","Data":"234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94"} Oct 30 00:50:19.868999 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.868974 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" event={"ID":"39dadf0c-1fee-4fec-9080-c5816e329cde","Type":"ContainerStarted","Data":"6b0e4ac015fe44b240c448fa9249a8d54f2e28803434a6239a009afe63a2de88"} Oct 30 00:50:19.912763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.908269 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" event={"ID":"e68d521f-b88b-40e6-8f15-f822d8394f55","Type":"ContainerStarted","Data":"eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5"} Oct 30 00:50:19.912763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.908344 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" event={"ID":"e68d521f-b88b-40e6-8f15-f822d8394f55","Type":"ContainerStarted","Data":"df85ff1f67fda9c3a4f10965c6e5f545cda17234015df531f7e58ec774d1bdd6"} Oct 30 00:50:19.924783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.914962 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podStartSLOduration=1.914942087 podStartE2EDuration="1.914942087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:19.914024805 +0000 UTC m=+957.826151949" watchObservedRunningTime="2026-10-30 00:50:19.914942087 +0000 UTC m=+957.827069230" Oct 30 00:50:19.980981 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:19.980901 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podStartSLOduration=1.980879234 podStartE2EDuration="1.980879234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:19.972340995 +0000 UTC m=+957.884468156" watchObservedRunningTime="2026-10-30 00:50:19.980879234 +0000 UTC m=+957.893006376" Oct 30 00:50:20.052377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.052308 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podStartSLOduration=2.052285216 podStartE2EDuration="2.052285216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:20.042112379 +0000 UTC m=+957.954239521" watchObservedRunningTime="2026-10-30 00:50:20.052285216 +0000 UTC m=+957.964412358" Oct 30 00:50:20.190170 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.190082 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:20.199900 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.199862 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:20.200165 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.200135 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:20.590637 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.590607 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:20.590969 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.590948 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:50:20.591156 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.591129 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 00:50:20.696143 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.696110 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:20.696862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.696835 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:50:20.697059 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:20.697033 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 00:50:21.189754 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.187496 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:21.189754 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.187560 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:21.220896 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.220839 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:21.247214 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.247161 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:21.265736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.264328 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:21.276621 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.276243 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 00:50:21.292101 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.292063 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:21.357880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.357834 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:21.413244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.412234 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:21.539112 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.539008 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:21.557868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.557817 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:50:21.587485 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.587452 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:50:21.587787 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.587761 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 00:50:21.593528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.593492 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:21.639836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.639784 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:50:21.676060 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.675693 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:21.696251 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.696121 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:50:21.696251 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.696188 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 00:50:21.761734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.751022 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:21.927039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:21.922831 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:22.188227 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.188125 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:22.188498 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.188452 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:22.223846 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.223801 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:22.224484 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.224457 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:50:22.230819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.230787 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:22.251933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.251852 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:50:22.280758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.265025 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:22.280758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.277227 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 00:50:22.326732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.293420 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:22.326732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.293543 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:50:22.326732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.293892 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:50:22.326732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.293945 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:50:22.326732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.293979 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:50:22.372736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.359802 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:22.372736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.360257 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:50:22.425737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.414658 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:22.425737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.416444 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:50:22.606868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.601270 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:50:22.626743 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.626397 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:50:22.626963 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.626941 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:50:22.652686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.628190 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:22.652686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.628510 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:50:22.669082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.655371 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:22.703862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:22.703822 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:50:23.060499 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.038032 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:50:23.060499 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.038100 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 00:50:23.060499 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.038198 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:50:23.060499 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.038239 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 00:50:23.267739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.264469 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:23.267739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.265869 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:50:23.267739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.266621 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:50:23.304582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.304441 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 00:50:23.343781 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.262942 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:23.343781 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.326629 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:50:23.354993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.353015 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:23.354993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.353077 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:50:23.354993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.354082 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:23.354993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.354128 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:23.354993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.354201 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:50:23.354993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.354656 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:50:23.366016 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.365985 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:23.442123 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.442066 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 00:50:23.623852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.613672 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:50:23.623852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.613747 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 00:50:23.623852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.615573 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:23.639377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.639322 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:23.707769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.705797 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:23.720932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.720897 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:50:23.721227 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:23.721186 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 00:50:24.038422 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.038384 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:24.063776 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.057355 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:24.237590 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.237557 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:24.237876 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.237848 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:24.243781 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.243750 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:24.307886 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.307790 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:24.330958 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.330920 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:24.365839 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.365792 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:24.366311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.366282 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:24.366723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.366679 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:50:24.468361 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.468330 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:50:24.515736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.490780 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:50:24.603993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.603896 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:50:24.610019 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.609981 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 00:50:24.615825 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.615782 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:24.624781 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.620853 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:50:24.720658 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.720609 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:50:24.737664 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.723681 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:24.741376 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.741350 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:50:24.741573 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.741560 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:50:24.784863 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.784827 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:50:24.785126 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.785098 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 00:50:24.913138 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:24.913054 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:50:25.297890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.297852 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:25.375190 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.375147 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:25.376524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.348889 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:25.382729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.379344 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:25.412627 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.386909 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:25.413774 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.340100 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:25.455570 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.455501 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:25.597261 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.595839 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:50:25.597261 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.597130 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 00:50:25.620892 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.620825 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:25.691301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.691178 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:50:25.691301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.691238 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 00:50:25.745496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.744673 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:25.761329 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.761286 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:25.782525 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.781797 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:25.871795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:25.871039 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:50:26.210840 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.208334 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:50:26.231735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.212173 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:26.231735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.212319 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:26.416738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.407217 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:50:26.417868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.417832 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:26.423884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.423216 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:26.599735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.595997 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:50:26.599735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.596052 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 00:50:26.600012 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.599780 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:26.601900 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.600161 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:50:26.626883 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.626846 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:26.697640 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.697606 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:50:26.697938 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.697907 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 00:50:26.854791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.850850 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:26.921898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:26.921838 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:27.018736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.018326 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:50:27.100087 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.099806 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:50:27.121905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.121000 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:50:27.227885 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.201349 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:27.227885 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.227340 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:27.285270 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.285215 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:27.296688 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.285640 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:27.345733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.344921 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:50:27.385790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.384789 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:50:27.389732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.384492 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:50:27.429739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.426098 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 00:50:27.601737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.591120 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:50:27.601737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.591182 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 00:50:27.691631 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.691532 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:50:27.691935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.691891 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 00:50:27.926884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.926820 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:27.927879 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.927838 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:50:27.947171 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.947076 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:50:27.948406 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:27.948386 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:50:28.188530 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:28.188490 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:28.188885 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:28.188848 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:28.234467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:28.234364 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:28.586813 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:28.586770 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:50:28.587027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:28.586836 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 00:50:28.698682 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:28.698647 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:28.699979 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:28.699954 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:28.701327 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:28.701303 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 00:50:29.187816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:29.187779 4517 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:50:29.188213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:29.187833 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:50:29.591023 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:29.590991 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:29.591451 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:29.591425 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:29.592216 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:29.592194 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 00:50:30.049290 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:30.049249 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:30.192035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:30.191999 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:30.192583 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:30.192554 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:30.193418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:30.193387 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 00:50:30.761493 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:50:30.761460 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:50:31.043815 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:31.043781 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:31.936604 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:31.936546 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:33.515967 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:33.515932 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:33.914407 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:33.914360 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:35.323497 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:35.323442 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:36.974836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:36.974803 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:38.782482 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:38.782444 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:40.500256 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:40.500214 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:42.102165 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:42.102132 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:43.843255 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:43.843208 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:46.025398 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.025351 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Oct 30 00:50:46.046074 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.046020 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded" Oct 30 00:50:46.059861 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.059807 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:46.101851 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.101800 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:46.136580 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.136515 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:46.317076 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.317008 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:46.349650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.349605 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:50:46.368460 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.358046 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:46.370187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.368344 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:46.890207 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:46.890154 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:47.062894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.062836 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:47.064461 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.064441 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:47.102897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.102842 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:47.104154 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.104135 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:50:47.142781 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.142672 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:50:47.143608 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.143586 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:50:47.388840 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.388731 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:47.626884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.626821 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:47.656806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.655868 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:47.953881 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:47.951801 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.076937 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.076874 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.162894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.162838 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.169851 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.169801 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.342145 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.342098 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:50:48.368848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.368807 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.382285 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.369222 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.402743 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.395370 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:50:48.454045 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.428803 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.456231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.455002 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:50:48.710292 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.708183 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.763898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.763860 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:50:48.768422 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.764132 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.862699 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.862669 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4"] Oct 30 00:50:48.986047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.985968 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:50:48.986837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.986569 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:48.996171 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:48.996143 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4"] Oct 30 00:50:49.017019 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.016974 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2"] Oct 30 00:50:49.043908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.043861 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2"] Oct 30 00:50:49.044342 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.044320 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.163367 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163327 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163385 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163407 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163426 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163457 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163496 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163512 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163539 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163568 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163585 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163603 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmkv\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-kube-api-access-vcmkv\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163630 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163662 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr64w\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-kube-api-access-hr64w\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163685 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163742 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163774 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.163887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.163826 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.164581 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.164525 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.265964 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266054 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266080 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266120 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266156 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266181 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266209 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmkv\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-kube-api-access-vcmkv\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266246 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266284 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr64w\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-kube-api-access-hr64w\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266316 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266365 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266397 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266442 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266505 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266565 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266640 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266682 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266749 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266778 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266848 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.266949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.266569 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.267471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.267108 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.267471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.267351 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.267471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.267361 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.267700 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.267682 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.267898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.267874 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.267976 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.267956 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.268026 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.267977 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.271741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.270181 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.271741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.270322 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.271741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.270436 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.271741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.270841 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.289499 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.289466 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmkv\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-kube-api-access-vcmkv\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.290139 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.290080 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.291842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.291820 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.294367 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.294344 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr64w\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-kube-api-access-hr64w\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.310491 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.310464 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:49.416947 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.416914 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:49.635350 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.635328 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4"] Oct 30 00:50:49.638093 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:50:49.638063 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7a6065_cd08_45df_b7ae_b3f42590dca7.slice/crio-f76cb5efc8168c6a27304e5fa6e0bd36af4965b614b5921641eaba3ee4b18e63 WatchSource:0}: Error finding container f76cb5efc8168c6a27304e5fa6e0bd36af4965b614b5921641eaba3ee4b18e63: Status 404 returned error can't find the container with id f76cb5efc8168c6a27304e5fa6e0bd36af4965b614b5921641eaba3ee4b18e63 Oct 30 00:50:49.640199 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.640156 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:49.640809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.640779 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:49.640864 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.640848 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:49.640907 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.640877 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:49.743758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.743724 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2"] Oct 30 00:50:49.750885 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:50:49.748366 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c730b4_39d7_4fc1_9b40_75d12dabc451.slice/crio-35acd924eea98501a617262ec8c8ec8ae1d1e0be161a968e61e5aea100b6d9ff WatchSource:0}: Error finding container 35acd924eea98501a617262ec8c8ec8ae1d1e0be161a968e61e5aea100b6d9ff: Status 404 returned error can't find the container with id 35acd924eea98501a617262ec8c8ec8ae1d1e0be161a968e61e5aea100b6d9ff Oct 30 00:50:49.753195 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.752377 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:49.753195 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.752524 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:49.753195 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.752574 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:50:49.952722 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.952388 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" event={"ID":"ac7a6065-cd08-45df-b7ae-b3f42590dca7","Type":"ContainerStarted","Data":"dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559"} Oct 30 00:50:49.952722 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.952463 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" event={"ID":"ac7a6065-cd08-45df-b7ae-b3f42590dca7","Type":"ContainerStarted","Data":"f76cb5efc8168c6a27304e5fa6e0bd36af4965b614b5921641eaba3ee4b18e63"} Oct 30 00:50:49.954891 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.954866 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" event={"ID":"a7c730b4-39d7-4fc1-9b40-75d12dabc451","Type":"ContainerStarted","Data":"6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5"} Oct 30 00:50:49.954997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:49.954917 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" event={"ID":"a7c730b4-39d7-4fc1-9b40-75d12dabc451","Type":"ContainerStarted","Data":"35acd924eea98501a617262ec8c8ec8ae1d1e0be161a968e61e5aea100b6d9ff"} Oct 30 00:50:50.000583 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:50.000516 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podStartSLOduration=2.000496179 podStartE2EDuration="2.000496179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:49.99725081 +0000 UTC m=+987.909377955" watchObservedRunningTime="2026-10-30 00:50:50.000496179 +0000 UTC m=+987.912623322" Oct 30 00:50:50.040943 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:50.040894 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podStartSLOduration=2.040876299 podStartE2EDuration="2.040876299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:50.038257994 +0000 UTC m=+987.950385188" watchObservedRunningTime="2026-10-30 00:50:50.040876299 +0000 UTC m=+987.953003441" Oct 30 00:50:50.311096 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:50.311064 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:50.312199 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:50.312178 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:50:50.312310 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:50.312231 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:50:50.417360 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:50.417322 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:50.418645 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:50.418624 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:50:50.418719 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:50.418677 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:50:51.314732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:51.313782 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:50:51.314732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:51.313843 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:50:51.373376 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:51.373313 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:51.420123 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:51.420086 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:50:51.420523 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:51.420472 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:50:52.314813 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:52.314778 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:50:52.315413 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:52.315380 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:50:52.444350 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:52.443835 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:50:52.444350 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:52.443906 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:50:53.124158 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:53.124121 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:53.328904 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:53.328869 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:50:53.329443 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:53.329408 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:50:53.422190 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:53.422100 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:50:53.422445 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:53.422422 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:50:54.313095 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:54.313055 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:50:54.313403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:54.313375 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:50:54.420870 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:54.420834 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:50:54.421493 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:54.421457 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:50:54.949352 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:54.949308 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:55.314462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:55.313016 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:50:55.314462 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:55.313086 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:50:55.419155 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:55.419120 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:50:55.419491 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:55.419441 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:50:56.311934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:56.311893 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:50:56.312356 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:56.311985 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:50:56.417882 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:56.417844 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:50:56.418088 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:56.417907 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:50:56.655225 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:56.655121 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:57.318518 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:57.318484 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:57.319276 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:57.319250 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:57.320064 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:57.320039 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 00:50:57.662321 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:57.662231 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:57.663914 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:57.663891 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:57.666333 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:57.666313 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 00:50:58.592264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:58.592223 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:58.710764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:50:58.710681 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:00.232632 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:00.232595 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:00.762674 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:51:00.762634 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:51:01.910528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:01.910472 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:03.630502 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:03.630474 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:05.227542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:05.227504 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:06.935081 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:06.935045 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:08.527542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:08.527510 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:10.755689 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:10.755648 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:12.084926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.084857 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.367043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.366860 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.382763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.380860 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.433274 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.433234 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.452363 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.452317 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.609763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.606831 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.632803 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.630803 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.654745 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.652796 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.785190 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.785142 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:12.785565 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:12.785437 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.339787 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:13.339740 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:51:13.378733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:13.365067 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.477361 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:13.421536 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.477361 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:13.428085 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:51:13.477361 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:13.457826 4517 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:13.477361 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:13.457883 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.859317 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:13.837242 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:13.859317 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:13.840864 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.274787 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.269916 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.285347 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.285313 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:14.351424 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.347678 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:51:14.385742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.366164 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.385742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.372185 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.385742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.373852 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:51:14.423745 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.421480 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.423745 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.422132 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:51:14.433052 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.432954 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:51:14.450455 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.450401 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:14.456750 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.453395 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:51:14.487404 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.487366 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.505755 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.502234 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:51:14.564221 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:14.564191 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:16.343373 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:16.343339 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:18.237890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.237852 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:18.750538 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.750507 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t"] Oct 30 00:51:18.754100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.754082 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:18.777224 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.777155 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:51:18.777224 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.777196 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:51:18.781487 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.781463 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t"] Oct 30 00:51:18.859673 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.859633 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4127e2-wsq2t\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:18.859895 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.859737 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4127e2-wsq2t\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:18.859968 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.859881 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55m8x\" (UniqueName: \"kubernetes.io/projected/0028d5f1-ba84-4517-981f-077ad1252946-kube-api-access-55m8x\") pod \"maas-api-revoke-sub-eb4127e2-wsq2t\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:18.961612 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.961568 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55m8x\" (UniqueName: \"kubernetes.io/projected/0028d5f1-ba84-4517-981f-077ad1252946-kube-api-access-55m8x\") pod \"maas-api-revoke-sub-eb4127e2-wsq2t\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:18.961842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.961671 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4127e2-wsq2t\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:18.961842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.961828 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4127e2-wsq2t\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:18.962447 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.962413 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4127e2-wsq2t\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:18.964905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.964882 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4127e2-wsq2t\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:18.983554 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:18.983519 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55m8x\" (UniqueName: \"kubernetes.io/projected/0028d5f1-ba84-4517-981f-077ad1252946-kube-api-access-55m8x\") pod \"maas-api-revoke-sub-eb4127e2-wsq2t\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:19.073671 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.073637 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:19.277941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.277907 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l"] Oct 30 00:51:19.282966 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.282942 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.310617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.310586 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l"] Oct 30 00:51:19.372595 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.372501 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.372595 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.372556 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.372595 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.372576 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.372932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.372600 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.372932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.372626 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.372932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.372643 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.372932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.372686 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6m7d\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-kube-api-access-r6m7d\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.372932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.372782 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.372932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.372848 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.393225 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.392490 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t"] Oct 30 00:51:19.470136 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.470107 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x"] Oct 30 00:51:19.474282 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474252 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6m7d\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-kube-api-access-r6m7d\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.474437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474318 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.474437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474381 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.474437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474416 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.474609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474453 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.474609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474470 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.474609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474494 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.474609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474517 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.474609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474533 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.474933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.474914 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.475394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.475371 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.476059 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.476040 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.476263 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.476238 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.476806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.476788 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.477316 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.477248 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.480224 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.480200 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.480524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.480500 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.506758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.505061 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6m7d\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-kube-api-access-r6m7d\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.509120 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.507950 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x"] Oct 30 00:51:19.509968 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.509931 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.575647 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.575611 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9pj\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-kube-api-access-4b9pj\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.575835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.575752 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.575835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.575807 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.575935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.575920 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.575999 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.575958 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.576057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.576034 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.576123 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.576101 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.576175 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.576162 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.576260 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.576241 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.603135 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.603105 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:19.677755 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.677691 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9pj\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-kube-api-access-4b9pj\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.677858 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.677783 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.677858 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.677822 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.677930 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.677894 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.677966 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.677931 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.678091 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.678073 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.678134 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.678110 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.678168 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.678160 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.678244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.678226 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.678320 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.678304 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.678757 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.678658 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.678910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.678865 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.679087 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.679018 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.679472 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.679448 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.681097 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.681069 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.682114 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.682092 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.700480 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.700442 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9pj\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-kube-api-access-4b9pj\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.700629 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.700559 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.794263 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.794225 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:19.988370 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:51:19.988340 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ef060c_8ab7_41a5_935b_8c7fa84559f7.slice/crio-7a4d87bdd6968872c08b2b558dd55442867c96a3ec05826aac758d3c7ff5895b WatchSource:0}: Error finding container 7a4d87bdd6968872c08b2b558dd55442867c96a3ec05826aac758d3c7ff5895b: Status 404 returned error can't find the container with id 7a4d87bdd6968872c08b2b558dd55442867c96a3ec05826aac758d3c7ff5895b Oct 30 00:51:19.989229 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.989197 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l"] Oct 30 00:51:19.991227 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.991189 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:51:19.991301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.991257 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:51:19.991301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:19.991287 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:51:20.118379 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.117868 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x"] Oct 30 00:51:20.124147 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.124111 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:51:20.124283 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.124203 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:51:20.124283 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.124247 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:51:20.173232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.173174 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:20.343354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.343248 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" event={"ID":"e3ef060c-8ab7-41a5-935b-8c7fa84559f7","Type":"ContainerStarted","Data":"50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468"} Oct 30 00:51:20.343354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.343305 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" event={"ID":"e3ef060c-8ab7-41a5-935b-8c7fa84559f7","Type":"ContainerStarted","Data":"7a4d87bdd6968872c08b2b558dd55442867c96a3ec05826aac758d3c7ff5895b"} Oct 30 00:51:20.346273 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.346238 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" event={"ID":"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4","Type":"ContainerStarted","Data":"d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706"} Oct 30 00:51:20.346428 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.346295 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" event={"ID":"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4","Type":"ContainerStarted","Data":"2a692e3349e5345c23512a4cf0d9e1956b2dc983aee7f4e80c2f7bfd2f079f9e"} Oct 30 00:51:20.348290 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.348258 4517 generic.go:356] "Generic (PLEG): container finished" podID="0028d5f1-ba84-4517-981f-077ad1252946" containerID="70f140750f4f67005b50658b01d2ec98ce279702e3a1aeab3111ac71aaf78322" exitCode=0 Oct 30 00:51:20.348421 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.348368 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" event={"ID":"0028d5f1-ba84-4517-981f-077ad1252946","Type":"ContainerDied","Data":"70f140750f4f67005b50658b01d2ec98ce279702e3a1aeab3111ac71aaf78322"} Oct 30 00:51:20.348421 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.348415 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" event={"ID":"0028d5f1-ba84-4517-981f-077ad1252946","Type":"ContainerStarted","Data":"3ecb718b2825abc7b7e56d01a16fc175bba0e637a32f066ebe9138516b099b9a"} Oct 30 00:51:20.388744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.388656 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" podStartSLOduration=1.388636883 podStartE2EDuration="1.388636883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:20.385291276 +0000 UTC m=+1018.297418431" watchObservedRunningTime="2026-10-30 00:51:20.388636883 +0000 UTC m=+1018.300764025" Oct 30 00:51:20.471737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.470887 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" podStartSLOduration=1.470867796 podStartE2EDuration="1.470867796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:20.462690141 +0000 UTC m=+1018.374817305" watchObservedRunningTime="2026-10-30 00:51:20.470867796 +0000 UTC m=+1018.382994937" Oct 30 00:51:20.603583 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.603493 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:20.803739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:20.795803 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:21.619823 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.615087 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:51:21.619823 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.615152 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:51:21.754084 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.754055 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:21.808731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.807956 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:51:21.808731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.808037 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:51:21.832207 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.832165 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0028d5f1-ba84-4517-981f-077ad1252946-kube-api-access-55m8x\" (UniqueName: \"kubernetes.io/projected/0028d5f1-ba84-4517-981f-077ad1252946-kube-api-access-55m8x\") pod \"0028d5f1-ba84-4517-981f-077ad1252946\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " Oct 30 00:51:21.832389 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.832267 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-openshift-ca\") pod \"0028d5f1-ba84-4517-981f-077ad1252946\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " Oct 30 00:51:21.832389 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.832309 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-xks-ca\") pod \"0028d5f1-ba84-4517-981f-077ad1252946\" (UID: \"0028d5f1-ba84-4517-981f-077ad1252946\") " Oct 30 00:51:21.837632 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.833459 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-openshift-ca" pod "0028d5f1-ba84-4517-981f-077ad1252946" (UID: "0028d5f1-ba84-4517-981f-077ad1252946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:21.837632 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.837312 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-xks-ca" pod "0028d5f1-ba84-4517-981f-077ad1252946" (UID: "0028d5f1-ba84-4517-981f-077ad1252946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:51:21.839152 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.838927 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0028d5f1-ba84-4517-981f-077ad1252946-kube-api-access-55m8x" pod "0028d5f1-ba84-4517-981f-077ad1252946" (UID: "0028d5f1-ba84-4517-981f-077ad1252946"). InnerVolumeSpecName "kube-api-access-55m8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:21.936816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.936691 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55m8x\" (UniqueName: \"kubernetes.io/projected/0028d5f1-ba84-4517-981f-077ad1252946-kube-api-access-55m8x\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:51:21.936816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.936744 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:51:21.936816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:21.936755 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0028d5f1-ba84-4517-981f-077ad1252946-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:51:22.389734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.387774 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" event={"ID":"0028d5f1-ba84-4517-981f-077ad1252946","Type":"ContainerDied","Data":"3ecb718b2825abc7b7e56d01a16fc175bba0e637a32f066ebe9138516b099b9a"} Oct 30 00:51:22.389734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.389303 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ecb718b2825abc7b7e56d01a16fc175bba0e637a32f066ebe9138516b099b9a" Oct 30 00:51:22.389734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.387836 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t" Oct 30 00:51:22.604167 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.604128 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:51:22.604459 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.604424 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:51:22.619159 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.618901 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct"] Oct 30 00:51:22.620767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.619639 4517 cpu_manager.go:414] "containerMap: removing container" podUID="0028d5f1-ba84-4517-981f-077ad1252946" containerName="revoke-keys" Oct 30 00:51:22.620767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.619664 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0028d5f1-ba84-4517-981f-077ad1252946" containerName="revoke-keys" Oct 30 00:51:22.621338 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.620944 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="0028d5f1-ba84-4517-981f-077ad1252946" containerName="revoke-keys" Oct 30 00:51:22.634912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.625138 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.640215 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.637890 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct"] Oct 30 00:51:22.640215 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.638788 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"distinct-2-w2-40ac41-kserve-self-signed-certs\"" Oct 30 00:51:22.646735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.645954 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-home\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.646735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.646018 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwhd2\" (UniqueName: \"kubernetes.io/projected/ec3f002e-af8a-40f2-a644-59662eaff603-kube-api-access-zwhd2\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.646735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.646050 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-model-cache\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.646735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.646116 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-tmp-dir\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.646735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.646131 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec3f002e-af8a-40f2-a644-59662eaff603-tls-certs\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.646735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.646224 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-dshm\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.738660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.738628 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:22.747448 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.747415 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-dshm\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.747621 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.747515 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-home\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.747621 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.747598 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwhd2\" (UniqueName: \"kubernetes.io/projected/ec3f002e-af8a-40f2-a644-59662eaff603-kube-api-access-zwhd2\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.747800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.747641 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-model-cache\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.747800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.747764 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-tmp-dir\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.747800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.747789 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec3f002e-af8a-40f2-a644-59662eaff603-tls-certs\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.748730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.748688 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-model-cache\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.749137 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.749118 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-home\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.749486 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.749470 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-tmp-dir\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.751449 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.751423 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec3f002e-af8a-40f2-a644-59662eaff603-tls-certs\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.751787 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.751766 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-dshm\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.779311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.779277 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwhd2\" (UniqueName: \"kubernetes.io/projected/ec3f002e-af8a-40f2-a644-59662eaff603-kube-api-access-zwhd2\") pod \"distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:22.798641 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.798177 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:51:22.798641 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.798236 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:51:22.962790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.938654 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:22.989742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:22.964932 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:23.605066 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:23.605030 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:51:23.614575 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:23.614486 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:51:23.651441 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:23.651409 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct"] Oct 30 00:51:23.656743 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:51:23.656691 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec3f002e_af8a_40f2_a644_59662eaff603.slice/crio-f65493fb99080a42b17930aa099b877c5b1861ece5c916dc707ac4621fe28893 WatchSource:0}: Error finding container f65493fb99080a42b17930aa099b877c5b1861ece5c916dc707ac4621fe28893: Status 404 returned error can't find the container with id f65493fb99080a42b17930aa099b877c5b1861ece5c916dc707ac4621fe28893 Oct 30 00:51:23.803733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:23.799742 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:51:23.803733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:23.799802 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:51:24.449638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:24.448682 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" event={"ID":"ec3f002e-af8a-40f2-a644-59662eaff603","Type":"ContainerStarted","Data":"252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838"} Oct 30 00:51:24.449638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:24.448757 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" event={"ID":"ec3f002e-af8a-40f2-a644-59662eaff603","Type":"ContainerStarted","Data":"f65493fb99080a42b17930aa099b877c5b1861ece5c916dc707ac4621fe28893"} Oct 30 00:51:24.517736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:24.516878 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podStartSLOduration=2.51685533 podStartE2EDuration="2.51685533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:24.490044826 +0000 UTC m=+1022.402171968" watchObservedRunningTime="2026-10-30 00:51:24.51685533 +0000 UTC m=+1022.428982472" Oct 30 00:51:24.628669 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:24.626828 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:51:24.628669 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:24.626885 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:51:24.801731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:24.800319 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:51:24.801731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:24.800397 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:51:24.977743 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:24.971803 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:25.522741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:25.522320 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:25.638300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:25.638266 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:51:25.638573 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:25.638542 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:51:25.796169 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:25.796128 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:51:25.796503 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:25.796467 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:51:26.603980 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:26.603935 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:51:26.604408 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:26.604006 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:51:26.795608 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:26.795568 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:51:26.795785 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:26.795643 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:51:28.030055 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:28.030015 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:28.031584 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:28.031559 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:28.032869 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:28.032847 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 00:51:28.126082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:28.126049 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:28.126923 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:28.126638 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:28.127452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:28.127429 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 00:51:28.154284 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:28.154253 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:30.247971 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:30.247938 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:30.768400 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:51:30.768346 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:51:31.305332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.305288 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4"] Oct 30 00:51:31.309854 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.309829 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.329459 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.329429 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4"] Oct 30 00:51:31.335079 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.335054 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:51:31.335792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.335276 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:51:31.437425 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.437391 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4aec633-n5sf4\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.437611 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.437455 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnplj\" (UniqueName: \"kubernetes.io/projected/b70a29be-83bc-4d79-8a25-92a23c802ebd-kube-api-access-fnplj\") pod \"maas-api-revoke-sub-f4aec633-n5sf4\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.437739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.437632 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4aec633-n5sf4\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.538885 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.538853 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4aec633-n5sf4\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.539072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.538915 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnplj\" (UniqueName: \"kubernetes.io/projected/b70a29be-83bc-4d79-8a25-92a23c802ebd-kube-api-access-fnplj\") pod \"maas-api-revoke-sub-f4aec633-n5sf4\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.539072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.538994 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4aec633-n5sf4\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.539633 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.539606 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4aec633-n5sf4\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.541459 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.541430 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4aec633-n5sf4\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.560941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.560873 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnplj\" (UniqueName: \"kubernetes.io/projected/b70a29be-83bc-4d79-8a25-92a23c802ebd-kube-api-access-fnplj\") pod \"maas-api-revoke-sub-f4aec633-n5sf4\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.622899 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.622868 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:31.928141 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:31.928116 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4"] Oct 30 00:51:31.930599 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:51:31.930562 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70a29be_83bc_4d79_8a25_92a23c802ebd.slice/crio-21b72bd04c9101d96c936565d9349449613b661bccce8bcfcacb6927e653eefb WatchSource:0}: Error finding container 21b72bd04c9101d96c936565d9349449613b661bccce8bcfcacb6927e653eefb: Status 404 returned error can't find the container with id 21b72bd04c9101d96c936565d9349449613b661bccce8bcfcacb6927e653eefb Oct 30 00:51:32.129283 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:32.129254 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:32.513867 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:32.513763 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" event={"ID":"b70a29be-83bc-4d79-8a25-92a23c802ebd","Type":"ContainerStarted","Data":"3efb3cc29accda34a4f57312da7ba59bc9165e3cae96ece9f8b3b3db6d603313"} Oct 30 00:51:32.514342 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:32.513877 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" event={"ID":"b70a29be-83bc-4d79-8a25-92a23c802ebd","Type":"ContainerStarted","Data":"21b72bd04c9101d96c936565d9349449613b661bccce8bcfcacb6927e653eefb"} Oct 30 00:51:32.553751 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:32.553688 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" podStartSLOduration=1.553676249 podStartE2EDuration="1.553676249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:32.552751388 +0000 UTC m=+1030.464878525" watchObservedRunningTime="2026-10-30 00:51:32.553676249 +0000 UTC m=+1030.465803390" Oct 30 00:51:32.965138 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:32.965108 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:32.978919 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:32.978890 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:33.520957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:33.520923 4517 generic.go:356] "Generic (PLEG): container finished" podID="b70a29be-83bc-4d79-8a25-92a23c802ebd" containerID="3efb3cc29accda34a4f57312da7ba59bc9165e3cae96ece9f8b3b3db6d603313" exitCode=0 Oct 30 00:51:33.521331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:33.521007 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" event={"ID":"b70a29be-83bc-4d79-8a25-92a23c802ebd","Type":"ContainerDied","Data":"3efb3cc29accda34a4f57312da7ba59bc9165e3cae96ece9f8b3b3db6d603313"} Oct 30 00:51:33.533197 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:33.533176 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:33.900602 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:33.900564 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:34.304932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.304901 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm"] Oct 30 00:51:34.309473 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.309448 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.336325 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.336292 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm"] Oct 30 00:51:34.386579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.375520 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.386579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.375814 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.386579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.375879 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.386579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.375934 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-kube-api-access-4tnt9\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.386579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.376006 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.386579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.376063 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.386579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.376186 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.386579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.376248 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.386579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.376291 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477155 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477125 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-kube-api-access-4tnt9\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477210 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477243 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477277 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477328 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477618 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477369 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477618 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477488 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477618 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477530 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477618 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477575 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.477878 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.477857 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.478051 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.478003 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.478154 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.478129 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.478263 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.478237 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.478377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.478358 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.480183 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.480155 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.484346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.484320 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.498687 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.498655 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.499232 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.499209 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-kube-api-access-4tnt9\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.624566 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.624532 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:34.700502 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.700170 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:34.781783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.781740 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-xks-ca\") pod \"b70a29be-83bc-4d79-8a25-92a23c802ebd\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " Oct 30 00:51:34.781997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.781856 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-openshift-ca\") pod \"b70a29be-83bc-4d79-8a25-92a23c802ebd\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " Oct 30 00:51:34.782073 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.781998 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70a29be-83bc-4d79-8a25-92a23c802ebd-kube-api-access-fnplj\" (UniqueName: \"kubernetes.io/projected/b70a29be-83bc-4d79-8a25-92a23c802ebd-kube-api-access-fnplj\") pod \"b70a29be-83bc-4d79-8a25-92a23c802ebd\" (UID: \"b70a29be-83bc-4d79-8a25-92a23c802ebd\") " Oct 30 00:51:34.782267 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.782241 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-openshift-ca" pod "b70a29be-83bc-4d79-8a25-92a23c802ebd" (UID: "b70a29be-83bc-4d79-8a25-92a23c802ebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:34.782922 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.782897 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:51:34.784242 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.784213 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70a29be-83bc-4d79-8a25-92a23c802ebd-kube-api-access-fnplj" pod "b70a29be-83bc-4d79-8a25-92a23c802ebd" (UID: "b70a29be-83bc-4d79-8a25-92a23c802ebd"). InnerVolumeSpecName "kube-api-access-fnplj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:34.784343 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.784261 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-xks-ca" pod "b70a29be-83bc-4d79-8a25-92a23c802ebd" (UID: "b70a29be-83bc-4d79-8a25-92a23c802ebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:51:34.889557 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.884925 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a29be-83bc-4d79-8a25-92a23c802ebd-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:51:34.889557 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.884990 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnplj\" (UniqueName: \"kubernetes.io/projected/b70a29be-83bc-4d79-8a25-92a23c802ebd-kube-api-access-fnplj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:51:34.965423 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.965388 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm"] Oct 30 00:51:34.966318 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:51:34.966291 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b676fb_dcc1_4e8d_bd05_5e33ac90d6dd.slice/crio-75441b73df22df9b1005621133736007164a7357709eba082982a21e47a9e35e WatchSource:0}: Error finding container 75441b73df22df9b1005621133736007164a7357709eba082982a21e47a9e35e: Status 404 returned error can't find the container with id 75441b73df22df9b1005621133736007164a7357709eba082982a21e47a9e35e Oct 30 00:51:34.969150 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.969111 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:51:34.969258 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.969195 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:51:34.969258 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:34.969238 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:51:35.535330 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:35.535302 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" Oct 30 00:51:35.535521 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:35.535294 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4" event={"ID":"b70a29be-83bc-4d79-8a25-92a23c802ebd","Type":"ContainerDied","Data":"21b72bd04c9101d96c936565d9349449613b661bccce8bcfcacb6927e653eefb"} Oct 30 00:51:35.535521 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:35.535423 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b72bd04c9101d96c936565d9349449613b661bccce8bcfcacb6927e653eefb" Oct 30 00:51:35.537048 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:35.537019 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" event={"ID":"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd","Type":"ContainerStarted","Data":"e2f5debeef1a62bfe7cf94905fedd5e002fc54b622bcfdc312ddd7a0274c9c64"} Oct 30 00:51:35.537155 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:35.537072 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" event={"ID":"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd","Type":"ContainerStarted","Data":"75441b73df22df9b1005621133736007164a7357709eba082982a21e47a9e35e"} Oct 30 00:51:35.580698 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:35.580633 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podStartSLOduration=1.5806112190000001 podStartE2EDuration="1.580611219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:35.577920126 +0000 UTC m=+1033.490047279" watchObservedRunningTime="2026-10-30 00:51:35.580611219 +0000 UTC m=+1033.492738361" Oct 30 00:51:35.624999 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:35.624957 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:35.645412 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:35.645371 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:36.626913 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:36.626871 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:51:36.627340 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:36.626931 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 00:51:37.594155 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:37.594118 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:37.625826 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:37.625793 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:51:37.626014 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:37.625852 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 00:51:38.625312 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:38.625278 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:51:38.625771 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:38.625332 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 00:51:39.624912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:39.624873 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:51:39.625146 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:39.624940 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 00:51:39.757314 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:39.757278 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:40.625547 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:40.625515 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:51:40.625898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:40.625580 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 00:51:41.644146 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:41.644107 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:51:41.646950 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:41.644726 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 00:51:42.418737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:42.416801 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.463732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:42.459861 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.540867 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:42.531189 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 00:51:42.626738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:42.626581 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.631733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:42.627891 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:51:42.631733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:42.627937 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 00:51:42.658747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:42.657075 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:51:43.221742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.207771 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:51:43.394907 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.381276 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.394907 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.383895 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.420134 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.417283 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.422168 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.420467 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:51:43.476153 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.474505 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:51:43.476153 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.475840 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:51:43.523046 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.515829 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.538384 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.537461 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.549101 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.545854 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.549101 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.546202 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:43.655739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.629732 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.655739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.652723 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:51:43.667649 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.660268 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.673776 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.672038 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:51:43.681360 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.681326 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:51:43.728187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.716863 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 00:51:44.005464 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:43.999551 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:51:44.015900 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.015855 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.181994 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.170757 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.245680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.240902 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:44.245680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.240964 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.253424 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.253354 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.305994 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.305874 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 30 00:51:44.384747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.382790 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:51:44.384747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.383199 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:51:44.385033 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.385008 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.388741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.385308 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:51:44.421299 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.420120 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.484849 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.484800 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:51:44.485547 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.485094 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.507852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.507804 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.517848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.517801 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.545176 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.545140 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:51:44.627798 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.546177 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 00:51:44.627798 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.604283 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:51:44.629041 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.629002 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 00:51:44.629206 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.629069 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 00:51:44.632732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.630123 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.683734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.666255 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.709818 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.708832 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.901791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.900242 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:51:44.997778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:44.980445 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:51:45.101732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.101549 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:51:45.399873 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.399829 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:45.400533 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.400158 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:45.428729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.421789 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:45.606483 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.606435 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:45.699108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.699008 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:51:45.728142 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.728095 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:45.756858 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.756813 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:45.760734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.757117 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:45.790971 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:45.790926 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:46.126737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.116262 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:51:46.126737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.118585 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.229855 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.229813 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.406226 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.406118 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.407586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.407552 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:51:46.436839 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.436789 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.493884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.493818 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:51:46.512090 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.512046 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:51:46.607915 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.607864 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.634863 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.634817 4517 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:46.635090 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.634895 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.701796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.700788 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.895303 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.894804 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.904428 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:46.903605 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:47.044737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.044173 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:51:47.089957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.087462 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:51:47.228057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.227872 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:51:47.229660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.229607 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:51:47.245543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.245515 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:51:47.252757 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.251465 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:51:47.256331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.255519 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:51:47.269501 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.269475 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:51:47.289926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.284375 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:47.289926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.285919 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:47.289926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.289120 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 00:51:47.479560 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:47.479334 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:50.632441 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:50.632406 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:52.366721 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:52.366684 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:54.005360 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:54.005317 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:55.600004 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:55.599970 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:57.313016 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:57.312964 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:59.181573 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.181536 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:59.353493 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.353461 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh"] Oct 30 00:51:59.353996 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.353981 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b70a29be-83bc-4d79-8a25-92a23c802ebd" containerName="revoke-keys" Oct 30 00:51:59.354050 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.353997 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70a29be-83bc-4d79-8a25-92a23c802ebd" containerName="revoke-keys" Oct 30 00:51:59.354100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.354091 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70a29be-83bc-4d79-8a25-92a23c802ebd" containerName="revoke-keys" Oct 30 00:51:59.358461 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.358421 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.373376 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.373351 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:51:59.384741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.383162 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:51:59.390322 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.390297 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh"] Oct 30 00:51:59.498370 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.498281 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7fe2f25-bxpfh\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.498517 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.498499 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdstl\" (UniqueName: \"kubernetes.io/projected/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-kube-api-access-mdstl\") pod \"maas-api-revoke-sub-d7fe2f25-bxpfh\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.498579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.498565 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7fe2f25-bxpfh\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.599790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.599752 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdstl\" (UniqueName: \"kubernetes.io/projected/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-kube-api-access-mdstl\") pod \"maas-api-revoke-sub-d7fe2f25-bxpfh\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.599987 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.599817 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7fe2f25-bxpfh\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.599987 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.599955 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7fe2f25-bxpfh\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.600512 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.600485 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7fe2f25-bxpfh\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.602772 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.602747 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7fe2f25-bxpfh\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.621807 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.621783 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdstl\" (UniqueName: \"kubernetes.io/projected/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-kube-api-access-mdstl\") pod \"maas-api-revoke-sub-d7fe2f25-bxpfh\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.678047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.678007 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:51:59.986101 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:51:59.986067 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh"] Oct 30 00:52:00.765881 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:52:00.765844 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:52:00.856369 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:00.856318 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:01.014627 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:01.014595 4517 generic.go:356] "Generic (PLEG): container finished" podID="9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1" containerID="f514650bf89a5f8ca7e76ca254edb6251e0a5bebe374a34199180624a0b3e339" exitCode=0 Oct 30 00:52:01.014860 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:01.014675 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" event={"ID":"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1","Type":"ContainerDied","Data":"f514650bf89a5f8ca7e76ca254edb6251e0a5bebe374a34199180624a0b3e339"} Oct 30 00:52:01.014860 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:01.014757 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" event={"ID":"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1","Type":"ContainerStarted","Data":"5a470a8a522e71a7f0d9b318f64bdfd1481b663194ab6be886ebcd562b0a4283"} Oct 30 00:52:02.182291 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.182261 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:52:02.344684 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.344653 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-kube-api-access-mdstl\" (UniqueName: \"kubernetes.io/projected/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-kube-api-access-mdstl\") pod \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " Oct 30 00:52:02.344881 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.344819 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-xks-ca\") pod \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " Oct 30 00:52:02.344937 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.344885 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-openshift-ca\") pod \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\" (UID: \"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1\") " Oct 30 00:52:02.345252 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.345229 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-openshift-ca" pod "9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1" (UID: "9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:52:02.345496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.345479 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:02.347037 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.347010 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-kube-api-access-mdstl" pod "9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1" (UID: "9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1"). InnerVolumeSpecName "kube-api-access-mdstl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:02.347246 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.347223 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-xks-ca" pod "9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1" (UID: "9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:52:02.447045 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.446996 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:02.447045 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.447034 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdstl\" (UniqueName: \"kubernetes.io/projected/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1-kube-api-access-mdstl\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:02.802670 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:02.802640 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:03.030059 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.030007 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" event={"ID":"9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1","Type":"ContainerDied","Data":"5a470a8a522e71a7f0d9b318f64bdfd1481b663194ab6be886ebcd562b0a4283"} Oct 30 00:52:03.030059 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.030043 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh" Oct 30 00:52:03.030302 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.030074 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a470a8a522e71a7f0d9b318f64bdfd1481b663194ab6be886ebcd562b0a4283" Oct 30 00:52:03.597491 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.597460 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt"] Oct 30 00:52:03.598308 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.598279 4517 cpu_manager.go:414] "containerMap: removing container" podUID="9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1" containerName="revoke-keys" Oct 30 00:52:03.598308 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.598305 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1" containerName="revoke-keys" Oct 30 00:52:03.598519 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.598428 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1" containerName="revoke-keys" Oct 30 00:52:03.602265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.602242 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.627948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.627912 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb"] Oct 30 00:52:03.644738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.632222 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt"] Oct 30 00:52:03.644738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.632331 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.682322 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.682296 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb"] Oct 30 00:52:03.774433 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774399 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.774599 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774460 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.774599 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774552 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwft\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-kube-api-access-fkwft\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.774599 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774585 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.774769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774605 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.774769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774623 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.774769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774746 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.774941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774806 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.774941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774847 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpfwv\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-kube-api-access-wpfwv\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.774941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.774889 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.775050 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.775023 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.775096 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.775077 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.775128 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.775109 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.775222 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.775205 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.775269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.775249 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.775377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.775363 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.775418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.775395 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5022b50c-515a-4ca3-af29-ee8ce01de949-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.775418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.775410 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.877292 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877186 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.877292 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877253 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.877535 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877304 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.877535 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877362 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.877535 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877399 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5022b50c-515a-4ca3-af29-ee8ce01de949-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.877535 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877506 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.877874 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877684 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.877874 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877781 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.878001 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877866 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwft\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-kube-api-access-fkwft\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.878001 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877902 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878001 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877933 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.878001 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.877961 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878001 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.878220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878040 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878072 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpfwv\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-kube-api-access-wpfwv\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878114 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878148 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878197 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.878220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878208 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878253 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.878586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878261 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5022b50c-515a-4ca3-af29-ee8ce01de949-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.878586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878299 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878523 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.878816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878653 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878653 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.878816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.878702 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.879207 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.879179 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.879355 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.879335 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.880125 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.880099 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.881136 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.881112 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.881303 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.881284 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.881457 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.881439 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.899522 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.899472 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.900800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.900773 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.900997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.900971 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwft\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-kube-api-access-fkwft\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:03.902723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.902683 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpfwv\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-kube-api-access-wpfwv\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:03.917132 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:03.917108 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:04.002464 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.002114 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:04.242039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.242016 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt"] Oct 30 00:52:04.243697 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:04.243669 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5022b50c_515a_4ca3_af29_ee8ce01de949.slice/crio-d419ab6949d394d3c708b5ec4c5360c8b79d36c8512bac1ef5afc4ead00c8a17 WatchSource:0}: Error finding container d419ab6949d394d3c708b5ec4c5360c8b79d36c8512bac1ef5afc4ead00c8a17: Status 404 returned error can't find the container with id d419ab6949d394d3c708b5ec4c5360c8b79d36c8512bac1ef5afc4ead00c8a17 Oct 30 00:52:04.246308 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.246275 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:04.246385 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.246344 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:04.246385 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.246377 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:04.340478 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.340367 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb"] Oct 30 00:52:04.343108 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:04.343076 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e883ea_a69a_4d3b_a80b_bc90b9edb040.slice/crio-50388e6b5e002ab1c11327137bd74454d9cef9122a709c7adbdf09d39af35210 WatchSource:0}: Error finding container 50388e6b5e002ab1c11327137bd74454d9cef9122a709c7adbdf09d39af35210: Status 404 returned error can't find the container with id 50388e6b5e002ab1c11327137bd74454d9cef9122a709c7adbdf09d39af35210 Oct 30 00:52:04.345878 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.345845 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:04.345989 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.345924 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:04.345989 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.345971 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:04.424073 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:04.424041 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:05.050497 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:05.050456 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" event={"ID":"5022b50c-515a-4ca3-af29-ee8ce01de949","Type":"ContainerStarted","Data":"7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b"} Oct 30 00:52:05.050974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:05.050518 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" event={"ID":"5022b50c-515a-4ca3-af29-ee8ce01de949","Type":"ContainerStarted","Data":"d419ab6949d394d3c708b5ec4c5360c8b79d36c8512bac1ef5afc4ead00c8a17"} Oct 30 00:52:05.052369 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:05.052343 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" event={"ID":"a8e883ea-a69a-4d3b-a80b-bc90b9edb040","Type":"ContainerStarted","Data":"6428260b307b3e34249bd58d23ebbbaab46768466ce1f35914b1e0e580d7e10a"} Oct 30 00:52:05.052512 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:05.052384 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" event={"ID":"a8e883ea-a69a-4d3b-a80b-bc90b9edb040","Type":"ContainerStarted","Data":"50388e6b5e002ab1c11327137bd74454d9cef9122a709c7adbdf09d39af35210"} Oct 30 00:52:05.126734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:05.122192 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podStartSLOduration=2.122172704 podStartE2EDuration="2.122172704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:05.108320351 +0000 UTC m=+1063.020447512" watchObservedRunningTime="2026-10-30 00:52:05.122172704 +0000 UTC m=+1063.034299845" Oct 30 00:52:05.155071 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:05.155018 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podStartSLOduration=2.155003319 podStartE2EDuration="2.155003319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:05.153204479 +0000 UTC m=+1063.065331621" watchObservedRunningTime="2026-10-30 00:52:05.155003319 +0000 UTC m=+1063.067130459" Oct 30 00:52:05.928732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:05.926440 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:05.928732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:05.926561 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:05.928732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:05.926604 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:06.018109 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:06.018060 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:06.057768 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:06.040963 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:06.057768 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:06.041020 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:06.687744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:06.677137 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:06.724733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:06.717790 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:52:06.899154 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:06.898104 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:06.927725 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:06.927668 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:06.928009 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:06.927975 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:06.928532 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:06.928499 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.013968 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.013875 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:07.014256 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.014218 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:07.305763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.302817 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.305763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.304685 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.332697 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.331355 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.411738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.410831 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.436736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.429808 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.528837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.528795 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.562244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.562146 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:52:07.611847 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.611797 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.678864 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.678806 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.726866 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.726809 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:52:07.728236 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.728209 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:52:07.936800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.920513 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:07.936800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.920575 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:07.945733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.939858 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 00:52:07.945733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:07.940252 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:52:08.109796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.082366 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.109796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.082702 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 00:52:08.180533 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.179386 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:08.180533 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.179452 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:08.333736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.319059 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:52:08.351767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.334357 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.458735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.426794 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.458735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.430239 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:52:08.458735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.430306 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:52:08.458735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.430342 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:52:08.458735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.448012 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:52:08.458735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.453564 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:52:08.543734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.537238 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.573734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.544500 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:52:08.574370 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.564874 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.593803 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.586161 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:52:08.673736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.632857 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.673736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.659425 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:52:08.699734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.698162 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.699734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.698577 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:52:08.802729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.793747 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 00:52:08.802729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.794294 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:52:08.926372 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.926278 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:08.928334 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.928296 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:08.940581 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:08.940538 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:09.010165 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.010126 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:09.020851 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.020805 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:09.127478 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.127432 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:09.157317 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.128016 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 00:52:09.203326 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.143320 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:52:09.203673 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.203649 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:52:09.203956 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.203904 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:52:09.204149 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.204135 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:52:09.227497 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.227466 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:52:09.320974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.318397 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:52:09.325497 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.325450 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:09.389125 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.389083 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:09.433754 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.432189 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:09.466300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.466206 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:52:09.548866 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.548814 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:52:09.613663 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.613618 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:52:09.675937 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.651322 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:09.711976 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.711934 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:09.930734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.921348 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:09.930734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:09.921465 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:10.020735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.019864 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:10.020735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.019975 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:10.169557 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.133647 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 00:52:10.230805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.203882 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:10.396321 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.329596 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:10.396321 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.330541 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:10.484063 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.483944 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:52:10.618735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.613843 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:52:10.752863 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.651192 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:10.752863 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.714972 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:10.752863 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.741801 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:10.752863 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.742260 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:10.752863 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.742342 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:52:10.929796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.929756 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:52:10.930394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.930042 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:10.931042 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:10.931013 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:11.015337 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.015240 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:11.015578 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.015548 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:11.015774 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.015751 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:11.041168 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.041134 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:52:11.165732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.163081 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:11.213509 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.212782 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 00:52:11.240597 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.234217 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:11.343289 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.343243 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:52:11.345530 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.345486 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:52:11.485861 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.485810 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:52:11.625195 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.625084 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:11.786886 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.786835 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:11.787987 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.787954 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:52:11.789273 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.789246 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:52:11.789672 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.789648 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:52:11.920925 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.920823 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:11.934075 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.934038 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:11.934356 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.934321 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:11.934494 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.934355 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:11.941680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:11.936440 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:52:12.012427 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.012394 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:12.012940 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.012908 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:12.016806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.016775 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.017291 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.017273 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:52:12.197794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.164968 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.303770 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.226807 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 00:52:12.303770 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.235835 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.303770 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.289285 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.390876 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.343997 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.390876 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.358275 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.503836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.476481 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:52:12.626529 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.528359 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.658335 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.656731 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.683435 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.677393 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:52:12.804834 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.788554 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.804834 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.790013 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:52:12.804834 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.800692 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.804834 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.801055 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:52:12.919296 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.917803 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:12.919296 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.917864 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:12.935738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.935172 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:52:12.947738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.947606 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:52:12.948338 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:12.948318 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:52:13.024036 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.024004 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:13.066930 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.051680 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:52:13.067622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.067591 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:13.192760 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.188959 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:52:13.226728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.226670 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:13.307044 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.307012 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:52:13.336528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.336442 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:52:13.337382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.337358 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:52:13.387206 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.387164 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:52:13.411629 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.411589 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:52:13.411948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.411931 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:52:13.439847 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.439803 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:13.455761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.452467 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:52:13.455761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.453733 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:52:13.486815 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.483212 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:13.553196 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.553159 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:52:13.918560 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.918521 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:13.918893 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:13.918862 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:14.008747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.008680 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:14.009215 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.009146 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:14.283794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.282739 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:14.519681 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.519635 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc"] Oct 30 00:52:14.524585 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.524563 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.535861 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.535831 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc"] Oct 30 00:52:14.538831 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.538675 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:52:14.542627 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.542287 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:52:14.644220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.644169 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52dc0096-crmgc\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.644612 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.644364 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52dc0096-crmgc\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.645015 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.644988 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2c2p\" (UniqueName: \"kubernetes.io/projected/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-kube-api-access-p2c2p\") pod \"maas-api-revoke-sub-52dc0096-crmgc\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.746273 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.746243 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52dc0096-crmgc\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.746619 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.746606 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2c2p\" (UniqueName: \"kubernetes.io/projected/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-kube-api-access-p2c2p\") pod \"maas-api-revoke-sub-52dc0096-crmgc\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.746786 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.746775 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52dc0096-crmgc\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.746967 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.746944 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52dc0096-crmgc\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.749397 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.749267 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52dc0096-crmgc\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.806316 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.806243 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2c2p\" (UniqueName: \"kubernetes.io/projected/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-kube-api-access-p2c2p\") pod \"maas-api-revoke-sub-52dc0096-crmgc\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.844451 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.844427 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:14.924581 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.924118 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:14.924581 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:14.924183 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:15.003586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.003540 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:15.003945 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.003887 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:15.032735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.032313 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8976-22de64"] Oct 30 00:52:15.037728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.037389 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8976-22de64" Oct 30 00:52:15.056775 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.056041 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v4hbx\"" Oct 30 00:52:15.069855 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.069445 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hspdh\" (UniqueName: \"kubernetes.io/projected/d39b6789-0432-4a9e-bfac-26b99b49de3a-kube-api-access-hspdh\") pod \"test-curl-8976-22de64\" (UID: \"d39b6789-0432-4a9e-bfac-26b99b49de3a\") " pod="opendatahub/test-curl-8976-22de64" Oct 30 00:52:15.080072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.080010 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8976-22de64"] Oct 30 00:52:15.171105 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.171065 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hspdh\" (UniqueName: \"kubernetes.io/projected/d39b6789-0432-4a9e-bfac-26b99b49de3a-kube-api-access-hspdh\") pod \"test-curl-8976-22de64\" (UID: \"d39b6789-0432-4a9e-bfac-26b99b49de3a\") " pod="opendatahub/test-curl-8976-22de64" Oct 30 00:52:15.194761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.194653 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hspdh\" (UniqueName: \"kubernetes.io/projected/d39b6789-0432-4a9e-bfac-26b99b49de3a-kube-api-access-hspdh\") pod \"test-curl-8976-22de64\" (UID: \"d39b6789-0432-4a9e-bfac-26b99b49de3a\") " pod="opendatahub/test-curl-8976-22de64" Oct 30 00:52:15.265057 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:15.265002 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bdd1c7_1d5f_4a00_8bd7_e2db7ac54a9b.slice/crio-e991ca1dbc096b183af3b1dc7b91d0bcc7411385b0f40de0c0ca6d63796b8131 WatchSource:0}: Error finding container e991ca1dbc096b183af3b1dc7b91d0bcc7411385b0f40de0c0ca6d63796b8131: Status 404 returned error can't find the container with id e991ca1dbc096b183af3b1dc7b91d0bcc7411385b0f40de0c0ca6d63796b8131 Oct 30 00:52:15.267752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.267725 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc"] Oct 30 00:52:15.377415 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.377329 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8976-22de64" Oct 30 00:52:15.437134 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.437098 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:15.702466 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.702435 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8976-22de64"] Oct 30 00:52:15.706838 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:15.706811 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd39b6789_0432_4a9e_bfac_26b99b49de3a.slice/crio-19b59b5882bb43ff9076599ceac00a60545a377ac99cbff0ff7cccc86f7634da WatchSource:0}: Error finding container 19b59b5882bb43ff9076599ceac00a60545a377ac99cbff0ff7cccc86f7634da: Status 404 returned error can't find the container with id 19b59b5882bb43ff9076599ceac00a60545a377ac99cbff0ff7cccc86f7634da Oct 30 00:52:15.709834 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.709805 4517 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:52:15.860679 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.860634 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8976-22de64" event={"ID":"d39b6789-0432-4a9e-bfac-26b99b49de3a","Type":"ContainerStarted","Data":"19b59b5882bb43ff9076599ceac00a60545a377ac99cbff0ff7cccc86f7634da"} Oct 30 00:52:15.863150 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.863121 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" event={"ID":"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b","Type":"ContainerStarted","Data":"c296bccca6c6e63f72fff4149b2de063da195001aad0def8d646dc36c9ec7858"} Oct 30 00:52:15.863305 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.863178 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" event={"ID":"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b","Type":"ContainerStarted","Data":"e991ca1dbc096b183af3b1dc7b91d0bcc7411385b0f40de0c0ca6d63796b8131"} Oct 30 00:52:15.903976 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.903922 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" podStartSLOduration=1.9039080400000001 podStartE2EDuration="1.90390804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:15.902007573 +0000 UTC m=+1073.814134738" watchObservedRunningTime="2026-10-30 00:52:15.90390804 +0000 UTC m=+1073.816035180" Oct 30 00:52:15.917817 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.917772 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:52:15.917954 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:15.917828 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:52:16.004034 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:16.003946 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:52:16.004034 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:16.003997 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:52:16.873076 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:16.873036 4517 generic.go:356] "Generic (PLEG): container finished" podID="70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b" containerID="c296bccca6c6e63f72fff4149b2de063da195001aad0def8d646dc36c9ec7858" exitCode=0 Oct 30 00:52:16.873579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:16.873254 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" event={"ID":"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b","Type":"ContainerDied","Data":"c296bccca6c6e63f72fff4149b2de063da195001aad0def8d646dc36c9ec7858"} Oct 30 00:52:16.923585 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:16.923555 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:16.924148 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:16.924088 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:16.925101 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:16.925075 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 00:52:17.008284 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:17.008255 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:17.008579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:17.008548 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:17.009652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:17.009631 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 00:52:17.349966 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:17.349908 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:17.630290 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:17.630209 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:17.756790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:17.756757 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:18.099318 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.099288 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:18.255696 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.255619 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-kube-api-access-p2c2p\" (UniqueName: \"kubernetes.io/projected/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-kube-api-access-p2c2p\") pod \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " Oct 30 00:52:18.255873 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.255776 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-xks-ca\") pod \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " Oct 30 00:52:18.255873 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.255806 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-openshift-ca\") pod \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\" (UID: \"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b\") " Oct 30 00:52:18.256297 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.256243 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-openshift-ca" pod "70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b" (UID: "70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:52:18.256453 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.256432 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:18.258012 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.257980 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-kube-api-access-p2c2p" pod "70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b" (UID: "70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b"). InnerVolumeSpecName "kube-api-access-p2c2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:18.258114 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.258014 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-xks-ca" pod "70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b" (UID: "70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:52:18.357651 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.357621 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2c2p\" (UniqueName: \"kubernetes.io/projected/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-kube-api-access-p2c2p\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:18.357651 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.357650 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:18.392088 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.392059 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:18.890338 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.890307 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" Oct 30 00:52:18.890338 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.890312 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc" event={"ID":"70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b","Type":"ContainerDied","Data":"e991ca1dbc096b183af3b1dc7b91d0bcc7411385b0f40de0c0ca6d63796b8131"} Oct 30 00:52:18.890576 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.890363 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e991ca1dbc096b183af3b1dc7b91d0bcc7411385b0f40de0c0ca6d63796b8131" Oct 30 00:52:18.892306 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.892276 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8976-22de64" event={"ID":"d39b6789-0432-4a9e-bfac-26b99b49de3a","Type":"ContainerStarted","Data":"5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1"} Oct 30 00:52:18.929425 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:18.929372 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8976-22de64" podStartSLOduration=1.326224385 podStartE2EDuration="3.929357279s" totalImagesPullingTime="2.603132894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:18.925650336 +0000 UTC m=+1076.837777458" watchObservedRunningTime="2026-10-30 00:52:18.929357279 +0000 UTC m=+1076.841484421" Oct 30 00:52:19.197451 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:19.197369 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:19.922304 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:19.922222 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8976-22de64"] Oct 30 00:52:19.927061 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:19.927033 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8976-22de64"] Oct 30 00:52:20.847441 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:20.847403 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:20.908132 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:20.908090 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8976-22de64" podUID="d39b6789-0432-4a9e-bfac-26b99b49de3a" containerName="test-curl-8976-22de64" containerID="cri-o://5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1" gracePeriod=2 Oct 30 00:52:20.966372 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:20.966337 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:21.212089 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:21.211946 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:21.712242 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:21.709457 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:22.012878 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:22.012790 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:22.839951 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:22.839918 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:23.198481 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.198455 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8976-22de64" Oct 30 00:52:23.352819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.352778 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39b6789-0432-4a9e-bfac-26b99b49de3a-kube-api-access-hspdh\" (UniqueName: \"kubernetes.io/projected/d39b6789-0432-4a9e-bfac-26b99b49de3a-kube-api-access-hspdh\") pod \"d39b6789-0432-4a9e-bfac-26b99b49de3a\" (UID: \"d39b6789-0432-4a9e-bfac-26b99b49de3a\") " Oct 30 00:52:23.355106 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.355078 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39b6789-0432-4a9e-bfac-26b99b49de3a-kube-api-access-hspdh" pod "d39b6789-0432-4a9e-bfac-26b99b49de3a" (UID: "d39b6789-0432-4a9e-bfac-26b99b49de3a"). InnerVolumeSpecName "kube-api-access-hspdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:23.454623 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.454539 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hspdh\" (UniqueName: \"kubernetes.io/projected/d39b6789-0432-4a9e-bfac-26b99b49de3a-kube-api-access-hspdh\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:23.575041 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.575005 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:23.935388 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.935354 4517 generic.go:356] "Generic (PLEG): container finished" podID="d39b6789-0432-4a9e-bfac-26b99b49de3a" containerID="5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1" exitCode=137 Oct 30 00:52:23.935616 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.935424 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8976-22de64" Oct 30 00:52:23.935616 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.935449 4517 scope.go:122] "RemoveContainer" containerID="5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1" Oct 30 00:52:23.946531 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.946508 4517 scope.go:122] "RemoveContainer" containerID="5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1" Oct 30 00:52:23.947043 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:52:23.947011 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1\": container with ID starting with 5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1 not found: ID does not exist" containerID="5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1" Oct 30 00:52:23.947260 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:23.947217 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1"} err="failed to get container status \"5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1\": rpc error: code = NotFound desc = could not find container \"5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1\": container with ID starting with 5c7e5f163e95ca2b220442ea04c69ba24d3497c1a20eee07ee893409e8f0abc1 not found: ID does not exist" Oct 30 00:52:23.959149 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:52:23.959112 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8976-22de64\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="d39b6789-0432-4a9e-bfac-26b99b49de3a" pod="opendatahub/test-curl-8976-22de64" Oct 30 00:52:24.116662 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.116629 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm"] Oct 30 00:52:24.117277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.117256 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d39b6789-0432-4a9e-bfac-26b99b49de3a" containerName="test-curl-8976-22de64" Oct 30 00:52:24.117382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.117279 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39b6789-0432-4a9e-bfac-26b99b49de3a" containerName="test-curl-8976-22de64" Oct 30 00:52:24.117382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.117320 4517 cpu_manager.go:414] "containerMap: removing container" podUID="70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b" containerName="revoke-keys" Oct 30 00:52:24.117382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.117328 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b" containerName="revoke-keys" Oct 30 00:52:24.117534 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.117487 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b" containerName="revoke-keys" Oct 30 00:52:24.117534 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.117508 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39b6789-0432-4a9e-bfac-26b99b49de3a" containerName="test-curl-8976-22de64" Oct 30 00:52:24.122391 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.122359 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.149886 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:52:24.149848 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8976-22de64\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="d39b6789-0432-4a9e-bfac-26b99b49de3a" pod="opendatahub/test-curl-8976-22de64" Oct 30 00:52:24.150324 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.150301 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-74b07396-openshift-default-dockercfg-dphwz\"" Oct 30 00:52:24.154870 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.154845 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm"] Oct 30 00:52:24.190467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.190386 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:24.206298 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.206262 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istiod-ca-cert\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.206865 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.206843 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-envoy\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.207066 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.207050 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l495p\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-kube-api-access-l495p\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.207299 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.207282 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-socket\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.207460 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.207446 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-token\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.207634 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.207617 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-credential-socket\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.207820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.207799 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-certs\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.207937 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.207853 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-data\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.207937 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.207875 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-podinfo\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.239602 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.239566 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39b6789-0432-4a9e-bfac-26b99b49de3a" path="/var/lib/kubelet/pods/d39b6789-0432-4a9e-bfac-26b99b49de3a/volumes" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.313586 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-socket\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.313669 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-token\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.313756 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-credential-socket\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.313801 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-certs\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.313867 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-data\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.313890 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-podinfo\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.314011 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istiod-ca-cert\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.314037 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-envoy\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.314075 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l495p\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-kube-api-access-l495p\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.314968 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-socket\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.315456 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-credential-socket\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.315734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.315690 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-certs\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.316557 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.315938 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-data\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.352731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.340675 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:24.352731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.341491 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istiod-ca-cert\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.352731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.345639 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-token\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.370890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.368581 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-podinfo\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.370890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.369455 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l495p\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-kube-api-access-l495p\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.370890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.369788 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-envoy\") pod \"e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.457795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.455363 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:24.832794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.832768 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm"] Oct 30 00:52:24.834010 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:24.833983 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bd9741_2ea2_4e2d_bdb0_25c486aab26f.slice/crio-d4e7c1f0f668d0eed4deb2768a8ec90f89c67747efddcb6918ccca86ac801240 WatchSource:0}: Error finding container d4e7c1f0f668d0eed4deb2768a8ec90f89c67747efddcb6918ccca86ac801240: Status 404 returned error can't find the container with id d4e7c1f0f668d0eed4deb2768a8ec90f89c67747efddcb6918ccca86ac801240 Oct 30 00:52:24.836582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.836543 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:24.836661 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.836631 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:24.836702 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.836676 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:24.954553 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.952521 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" event={"ID":"84bd9741-2ea2-4e2d-bdb0-25c486aab26f","Type":"ContainerStarted","Data":"77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75"} Oct 30 00:52:24.954553 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.952580 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" event={"ID":"84bd9741-2ea2-4e2d-bdb0-25c486aab26f","Type":"ContainerStarted","Data":"d4e7c1f0f668d0eed4deb2768a8ec90f89c67747efddcb6918ccca86ac801240"} Oct 30 00:52:24.962987 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.962956 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:24.995811 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:24.995753 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" podStartSLOduration=0.99573418 podStartE2EDuration="995.73418ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:24.995095049 +0000 UTC m=+1082.907222192" watchObservedRunningTime="2026-10-30 00:52:24.99573418 +0000 UTC m=+1082.907861340" Oct 30 00:52:25.084455 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.084352 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:25.386910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.386807 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:25.458323 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.458286 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:25.459550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.459521 4517 patch_prober.go:29] interesting pod/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:52:25.459733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.459584 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" podUID="84bd9741-2ea2-4e2d-bdb0-25c486aab26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 00:52:25.520959 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.520912 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:25.633084 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.633049 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:25.833166 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.833118 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4"] Oct 30 00:52:25.838035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.838011 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.872079 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.872038 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c4e6c558-openshift-default-dockercfg-mqd2n\"" Oct 30 00:52:25.884077 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.884036 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4"] Oct 30 00:52:25.946656 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.946616 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-socket\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.946894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.946673 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-data\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.946978 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.946922 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-podinfo\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.947036 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.946976 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knbh4\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-kube-api-access-knbh4\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.947097 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.947071 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-envoy\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.947154 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.947113 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4a30d0-d849-41cf-be6c-de15b996862a-istiod-ca-cert\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.947264 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.947245 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-certs\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.947329 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.947308 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-credential-socket\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.947387 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.947341 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-token\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:25.980657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:25.980596 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:26.049748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.049661 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-socket\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.049948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.049756 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-data\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.049948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.049884 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-podinfo\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.049948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.049923 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knbh4\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-kube-api-access-knbh4\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.050131 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.049981 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-envoy\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.050131 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.050011 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4a30d0-d849-41cf-be6c-de15b996862a-istiod-ca-cert\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.050242 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.050179 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-certs\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.050384 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.050252 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-credential-socket\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.050384 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.050285 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-token\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.052062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.051814 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4a30d0-d849-41cf-be6c-de15b996862a-istiod-ca-cert\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.052390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.052363 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-credential-socket\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.052621 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.052588 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-socket\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.052861 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.052844 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-data\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.055739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.053920 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-certs\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.056995 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.056306 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-podinfo\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.076821 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.076786 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-envoy\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.077029 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.077006 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knbh4\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-kube-api-access-knbh4\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.077401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.077381 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-token\") pod \"e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.110788 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.109851 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:26.181732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.181094 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:26.469576 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.468988 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:26.579817 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.579779 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4"] Oct 30 00:52:26.582269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.582230 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:26.593240 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.593197 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:26.593482 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.593457 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:26.593638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.593618 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:26.703104 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.703069 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:26.838491 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.838453 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst"] Oct 30 00:52:26.843616 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.843588 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:26.875756 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.867582 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst"] Oct 30 00:52:26.875756 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.874246 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:52:26.875756 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:26.874319 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:52:27.092888 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.092095 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b18da0-vqqst\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.093406 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.093385 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqtgm\" (UniqueName: \"kubernetes.io/projected/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-kube-api-access-jqtgm\") pod \"maas-api-revoke-sub-37b18da0-vqqst\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.093604 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.093589 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b18da0-vqqst\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.129848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.122901 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" event={"ID":"8b4a30d0-d849-41cf-be6c-de15b996862a","Type":"ContainerStarted","Data":"b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a"} Oct 30 00:52:27.129848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.122973 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" event={"ID":"8b4a30d0-d849-41cf-be6c-de15b996862a","Type":"ContainerStarted","Data":"d52c2749848c56c2268ac26f95e8507f6325b143a3a5373f610331f05d021d57"} Oct 30 00:52:27.129848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.123003 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:27.129848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.125183 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:52:27.175734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.172756 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" podStartSLOduration=2.172735093 podStartE2EDuration="2.172735093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:27.167223684 +0000 UTC m=+1085.079350826" watchObservedRunningTime="2026-10-30 00:52:27.172735093 +0000 UTC m=+1085.084862252" Oct 30 00:52:27.183072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.183020 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:27.186217 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.186184 4517 patch_prober.go:29] interesting pod/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 00:52:27.186392 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.186240 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" podUID="8b4a30d0-d849-41cf-be6c-de15b996862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 00:52:27.196389 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.195209 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqtgm\" (UniqueName: \"kubernetes.io/projected/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-kube-api-access-jqtgm\") pod \"maas-api-revoke-sub-37b18da0-vqqst\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.196389 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.195272 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b18da0-vqqst\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.196389 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.195522 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b18da0-vqqst\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.196745 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.196689 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b18da0-vqqst\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.201222 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.200510 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b18da0-vqqst\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.244386 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.241285 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqtgm\" (UniqueName: \"kubernetes.io/projected/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-kube-api-access-jqtgm\") pod \"maas-api-revoke-sub-37b18da0-vqqst\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.461837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.461728 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:27.595600 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.592825 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:27.736126 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.736009 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:27.810195 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.810171 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst"] Oct 30 00:52:27.812866 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:27.812833 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3dcf77_6987_443d_bd45_ba998e1f2cc6.slice/crio-e1b9b4b4ad9998c38b8da03e19efeb92d53adabbdb840197bf99149b923709f8 WatchSource:0}: Error finding container e1b9b4b4ad9998c38b8da03e19efeb92d53adabbdb840197bf99149b923709f8: Status 404 returned error can't find the container with id e1b9b4b4ad9998c38b8da03e19efeb92d53adabbdb840197bf99149b923709f8 Oct 30 00:52:27.862316 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:27.862282 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:28.135420 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:28.135371 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" event={"ID":"fd3dcf77-6987-443d-bd45-ba998e1f2cc6","Type":"ContainerStarted","Data":"e1b9b4b4ad9998c38b8da03e19efeb92d53adabbdb840197bf99149b923709f8"} Oct 30 00:52:28.189342 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:28.189316 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:28.391701 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:28.391582 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:29.201730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:29.196811 4517 generic.go:356] "Generic (PLEG): container finished" podID="fd3dcf77-6987-443d-bd45-ba998e1f2cc6" containerID="2f33f70e04ce59c192a1efb2dd135fc33ceeac0b86e6fb2c977401240bdd5ffa" exitCode=0 Oct 30 00:52:29.201730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:29.198095 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" event={"ID":"fd3dcf77-6987-443d-bd45-ba998e1f2cc6","Type":"ContainerDied","Data":"2f33f70e04ce59c192a1efb2dd135fc33ceeac0b86e6fb2c977401240bdd5ffa"} Oct 30 00:52:29.201730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:29.199197 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:29.214856 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:29.214830 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:52:29.717568 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:29.717513 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:29.737732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:29.718072 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:29.860790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:29.860749 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:30.158865 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.158827 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:30.159333 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.159307 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 00:52:30.182729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.169799 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.225879 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.225813 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.359403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.319038 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.359403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.344603 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.359403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.345183 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:30.450789 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.448969 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.505952 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.496663 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.599465 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.599426 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:30.680733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.660810 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.768791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.753882 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded" Oct 30 00:52:30.805734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:30.804080 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.258457 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.251937 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.258457 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.252752 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:52:31.320978 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:52:31.320940 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:52:31.321426 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.321392 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.322261 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.322239 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:52:31.386007 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.367597 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.386007 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.381170 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:52:31.386007 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.382162 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.416957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.416905 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.490874 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.479837 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.490874 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.480256 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:52:31.548542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.505792 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.548542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.506223 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:52:31.677786 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.662522 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.677786 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.663263 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:52:31.777892 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.777849 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.830783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.806273 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.830783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.820075 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:52:31.944733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.919809 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.948837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:31.948806 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:52:32.183950 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.183834 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 00:52:32.187897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.187870 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:32.286735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.285792 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.286735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.286137 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.355748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.349130 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.393572 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.392143 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.393572 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.392421 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.455790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.393962 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-openshift-ca\") pod \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " Oct 30 00:52:32.455790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.394128 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-xks-ca\") pod \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " Oct 30 00:52:32.455790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.394177 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-kube-api-access-jqtgm\" (UniqueName: \"kubernetes.io/projected/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-kube-api-access-jqtgm\") pod \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\" (UID: \"fd3dcf77-6987-443d-bd45-ba998e1f2cc6\") " Oct 30 00:52:32.455790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.447051 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-openshift-ca" pod "fd3dcf77-6987-443d-bd45-ba998e1f2cc6" (UID: "fd3dcf77-6987-443d-bd45-ba998e1f2cc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:52:32.456154 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.456083 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.525733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.525223 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.561544 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.527558 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.561544 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.539337 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:32.623149 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.623107 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.624515 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.624492 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:52:32.624750 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.624734 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:52:32.657144 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.657103 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.670205 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.669822 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.703209 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.703168 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-xks-ca" pod "fd3dcf77-6987-443d-bd45-ba998e1f2cc6" (UID: "fd3dcf77-6987-443d-bd45-ba998e1f2cc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:52:32.711226 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.711146 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:52:32.764532 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.764501 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:32.787738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.767909 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-kube-api-access-jqtgm" pod "fd3dcf77-6987-443d-bd45-ba998e1f2cc6" (UID: "fd3dcf77-6987-443d-bd45-ba998e1f2cc6"). InnerVolumeSpecName "kube-api-access-jqtgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:32.788018 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.787984 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.788528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.788505 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:52:32.889734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:32.867867 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqtgm\" (UniqueName: \"kubernetes.io/projected/fd3dcf77-6987-443d-bd45-ba998e1f2cc6-kube-api-access-jqtgm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:33.134392 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.134359 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:52:33.222692 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.222653 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:52:33.224007 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.223967 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:33.343407 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.227781 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:33.344111 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.337730 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:52:33.345359 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.345327 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 00:52:33.354933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.354764 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 00:52:33.426792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.424848 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:33.530620 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.530466 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:52:33.553166 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.553103 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" event={"ID":"fd3dcf77-6987-443d-bd45-ba998e1f2cc6","Type":"ContainerDied","Data":"e1b9b4b4ad9998c38b8da03e19efeb92d53adabbdb840197bf99149b923709f8"} Oct 30 00:52:33.582815 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.582782 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1b9b4b4ad9998c38b8da03e19efeb92d53adabbdb840197bf99149b923709f8" Oct 30 00:52:33.610604 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.596410 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst" Oct 30 00:52:33.610934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.599025 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:33.638273 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.615866 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:52:33.663076 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.663049 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:52:33.663311 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.663294 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:52:33.762400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.762317 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:52:33.789149 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.788490 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:52:33.816471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.816444 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:52:33.841928 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.826621 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw"] Oct 30 00:52:33.841928 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.827494 4517 cpu_manager.go:414] "containerMap: removing container" podUID="fd3dcf77-6987-443d-bd45-ba998e1f2cc6" containerName="revoke-keys" Oct 30 00:52:33.841928 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.827515 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3dcf77-6987-443d-bd45-ba998e1f2cc6" containerName="revoke-keys" Oct 30 00:52:33.841928 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.827736 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3dcf77-6987-443d-bd45-ba998e1f2cc6" containerName="revoke-keys" Oct 30 00:52:33.846160 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.846132 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:33.851000 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.849702 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m"] Oct 30 00:52:33.873930 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.873901 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw"] Oct 30 00:52:33.874162 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.874147 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:52:33.874306 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.874294 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m"] Oct 30 00:52:33.874518 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.874498 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:33.950471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.948313 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqcd4\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-kube-api-access-dqcd4\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:33.950471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.948456 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:33.950471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.948491 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:33.950471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.948583 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:33.950471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.948687 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b3ba8-8192-4bf6-8a03-846749be71c5-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:33.950471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.948814 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:33.950471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.948905 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:33.950471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.948930 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:33.950471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:33.948984 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.013522 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.013408 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:34.050173 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050145 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.050320 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050180 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.050320 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050232 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.050320 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050265 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.050320 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050295 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.050533 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050325 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.050533 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050367 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.050533 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050391 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.050533 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050450 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.050533 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050494 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqcd4\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-kube-api-access-dqcd4\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.050815 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050589 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.050815 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050660 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.050815 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050689 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.050815 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050767 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.050975 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.050845 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.051161 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.051136 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.051265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.051242 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b3ba8-8192-4bf6-8a03-846749be71c5-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.051374 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.051355 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ftn\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-kube-api-access-l2ftn\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.051442 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.051402 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.051550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.051527 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.051936 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.051917 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.052550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.052529 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.052677 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.052577 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b3ba8-8192-4bf6-8a03-846749be71c5-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.054890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.054867 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.055038 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.055017 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.081139 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.081087 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqcd4\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-kube-api-access-dqcd4\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.081877 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.081852 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.151392 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.139111 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:34.156315 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.156276 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.156837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.156381 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.156837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.156543 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ftn\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-kube-api-access-l2ftn\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.156837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.156582 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.156837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.156618 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.156837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.156667 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.156837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.156700 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.156837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.156785 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.156837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.156816 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.161341 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.161313 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.161493 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.161411 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.161602 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.161419 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.162054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.162028 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.162167 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.162151 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.163003 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.162898 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.165724 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.165685 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.187088 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.179849 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:34.200430 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.198258 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.200430 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.198964 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ftn\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-kube-api-access-l2ftn\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.215976 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.215946 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:34.516582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.516551 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw"] Oct 30 00:52:34.517334 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:34.517310 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325b3ba8_8192_4bf6_8a03_846749be71c5.slice/crio-3b560638c2377506f13c1c6d94318c639355b0e6557599d3f8094b9e72d4cfb4 WatchSource:0}: Error finding container 3b560638c2377506f13c1c6d94318c639355b0e6557599d3f8094b9e72d4cfb4: Status 404 returned error can't find the container with id 3b560638c2377506f13c1c6d94318c639355b0e6557599d3f8094b9e72d4cfb4 Oct 30 00:52:34.519852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.519819 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:34.519914 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.519886 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:34.519946 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.519919 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:34.550742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.549189 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" event={"ID":"325b3ba8-8192-4bf6-8a03-846749be71c5","Type":"ContainerStarted","Data":"3b560638c2377506f13c1c6d94318c639355b0e6557599d3f8094b9e72d4cfb4"} Oct 30 00:52:34.565527 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.565503 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m"] Oct 30 00:52:34.567907 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:34.567877 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c81e6f4_4b34_4711_ba3f_09be57119c0a.slice/crio-3ddd6f5d4cd28b8886e1b6e5395c4e90b05f7c68114a194562925bee770df303 WatchSource:0}: Error finding container 3ddd6f5d4cd28b8886e1b6e5395c4e90b05f7c68114a194562925bee770df303: Status 404 returned error can't find the container with id 3ddd6f5d4cd28b8886e1b6e5395c4e90b05f7c68114a194562925bee770df303 Oct 30 00:52:34.571047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.571005 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:34.571164 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.571093 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:34.571258 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.571163 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:34.653091 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:34.653056 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:35.557887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:35.557856 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" event={"ID":"325b3ba8-8192-4bf6-8a03-846749be71c5","Type":"ContainerStarted","Data":"d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84"} Oct 30 00:52:35.559895 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:35.559866 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" event={"ID":"2c81e6f4-4b34-4711-ba3f-09be57119c0a","Type":"ContainerStarted","Data":"a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b"} Oct 30 00:52:35.560047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:35.559922 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" event={"ID":"2c81e6f4-4b34-4711-ba3f-09be57119c0a","Type":"ContainerStarted","Data":"3ddd6f5d4cd28b8886e1b6e5395c4e90b05f7c68114a194562925bee770df303"} Oct 30 00:52:35.612334 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:35.612264 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" podStartSLOduration=2.612245837 podStartE2EDuration="2.612245837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:35.60904389 +0000 UTC m=+1093.521171031" watchObservedRunningTime="2026-10-30 00:52:35.612245837 +0000 UTC m=+1093.524372979" Oct 30 00:52:35.653059 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:35.653012 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" podStartSLOduration=2.652998455 podStartE2EDuration="2.652998455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:35.649317475 +0000 UTC m=+1093.561444615" watchObservedRunningTime="2026-10-30 00:52:35.652998455 +0000 UTC m=+1093.565125596" Oct 30 00:52:36.186788 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:36.186066 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:36.203732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:36.199878 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:52:36.203732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:36.199952 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 00:52:36.218301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:36.218222 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:36.218584 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:36.218562 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:52:36.218805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:36.218780 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 00:52:36.950198 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:36.949321 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:37.188735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.182197 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:52:37.188735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.182258 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 00:52:37.221799 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.218974 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:52:37.221799 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.219043 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 00:52:37.461736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.457796 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph"] Oct 30 00:52:37.476448 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.476368 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.511897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.510987 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 00:52:37.511897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.511055 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:52:37.514177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.514150 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph"] Oct 30 00:52:37.551652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.551621 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d0591d4-x74ph\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.551909 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.551895 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d0591d4-x74ph\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.552097 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.552074 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfkg\" (UniqueName: \"kubernetes.io/projected/b9096fe6-7e29-418d-97be-ded94d123693-kube-api-access-nlfkg\") pod \"maas-api-revoke-sub-0d0591d4-x74ph\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.653830 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.653802 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d0591d4-x74ph\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.653991 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.653895 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d0591d4-x74ph\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.654046 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.654022 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfkg\" (UniqueName: \"kubernetes.io/projected/b9096fe6-7e29-418d-97be-ded94d123693-kube-api-access-nlfkg\") pod \"maas-api-revoke-sub-0d0591d4-x74ph\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.654503 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.654477 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d0591d4-x74ph\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.656740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.656302 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d0591d4-x74ph\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.687794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.685358 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfkg\" (UniqueName: \"kubernetes.io/projected/b9096fe6-7e29-418d-97be-ded94d123693-kube-api-access-nlfkg\") pod \"maas-api-revoke-sub-0d0591d4-x74ph\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:37.853869 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:37.853839 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:38.202734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:38.196373 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:52:38.202734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:38.196429 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 00:52:38.220734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:38.219806 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:52:38.220734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:38.219884 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 00:52:38.424504 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:38.424401 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9096fe6_7e29_418d_97be_ded94d123693.slice/crio-fbb38bb853b30477abaa7cd0f45d2c6d5499c001d08f08c3957ee4ac64458073 WatchSource:0}: Error finding container fbb38bb853b30477abaa7cd0f45d2c6d5499c001d08f08c3957ee4ac64458073: Status 404 returned error can't find the container with id fbb38bb853b30477abaa7cd0f45d2c6d5499c001d08f08c3957ee4ac64458073 Oct 30 00:52:38.424504 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:38.424494 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:38.431284 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:38.431259 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph"] Oct 30 00:52:38.541562 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:38.541522 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:38.641845 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:38.641764 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" event={"ID":"b9096fe6-7e29-418d-97be-ded94d123693","Type":"ContainerStarted","Data":"fbb38bb853b30477abaa7cd0f45d2c6d5499c001d08f08c3957ee4ac64458073"} Oct 30 00:52:39.181039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:39.181000 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:52:39.181351 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:39.181309 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 00:52:39.223476 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:39.223437 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:52:39.224039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:39.224007 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 00:52:39.309734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:39.300192 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:39.426178 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:39.426139 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:39.559730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:39.555107 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:39.719794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:39.710999 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" event={"ID":"b9096fe6-7e29-418d-97be-ded94d123693","Type":"ContainerStarted","Data":"97aa5ed0ed520b79d2ac4e420ca4e62274006f47b31d13d41c87fbb9da26798c"} Oct 30 00:52:39.788737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:39.785304 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" podStartSLOduration=2.785285422 podStartE2EDuration="2.785285422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:39.769792939 +0000 UTC m=+1097.681920073" watchObservedRunningTime="2026-10-30 00:52:39.785285422 +0000 UTC m=+1097.697412560" Oct 30 00:52:40.186736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:40.186155 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:52:40.186736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:40.186228 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 00:52:40.219772 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:40.219149 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:52:40.219772 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:40.219204 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 00:52:40.713740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:40.698772 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:40.728769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:40.728732 4517 generic.go:356] "Generic (PLEG): container finished" podID="b9096fe6-7e29-418d-97be-ded94d123693" containerID="97aa5ed0ed520b79d2ac4e420ca4e62274006f47b31d13d41c87fbb9da26798c" exitCode=0 Oct 30 00:52:40.728952 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:40.728834 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" event={"ID":"b9096fe6-7e29-418d-97be-ded94d123693","Type":"ContainerDied","Data":"97aa5ed0ed520b79d2ac4e420ca4e62274006f47b31d13d41c87fbb9da26798c"} Oct 30 00:52:40.827948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:40.827843 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:40.945408 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:40.945373 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:41.186135 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:41.182500 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:52:41.186135 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:41.182560 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 00:52:41.218181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:41.218024 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:52:41.218181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:41.218081 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 00:52:41.329230 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:41.329188 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:41.929622 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:41.929589 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:42.055521 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.055486 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9096fe6-7e29-418d-97be-ded94d123693-kube-api-access-nlfkg\" (UniqueName: \"kubernetes.io/projected/b9096fe6-7e29-418d-97be-ded94d123693-kube-api-access-nlfkg\") pod \"b9096fe6-7e29-418d-97be-ded94d123693\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " Oct 30 00:52:42.055811 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.055773 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-openshift-ca\") pod \"b9096fe6-7e29-418d-97be-ded94d123693\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " Oct 30 00:52:42.055811 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.055827 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-xks-ca\") pod \"b9096fe6-7e29-418d-97be-ded94d123693\" (UID: \"b9096fe6-7e29-418d-97be-ded94d123693\") " Oct 30 00:52:42.056585 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.056412 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-openshift-ca" pod "b9096fe6-7e29-418d-97be-ded94d123693" (UID: "b9096fe6-7e29-418d-97be-ded94d123693"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:52:42.056862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.056839 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:42.058075 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.058042 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9096fe6-7e29-418d-97be-ded94d123693-kube-api-access-nlfkg" pod "b9096fe6-7e29-418d-97be-ded94d123693" (UID: "b9096fe6-7e29-418d-97be-ded94d123693"). InnerVolumeSpecName "kube-api-access-nlfkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:42.058492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.058472 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-xks-ca" pod "b9096fe6-7e29-418d-97be-ded94d123693" (UID: "b9096fe6-7e29-418d-97be-ded94d123693"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:52:42.158870 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.158834 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096fe6-7e29-418d-97be-ded94d123693-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:42.158870 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.158867 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlfkg\" (UniqueName: \"kubernetes.io/projected/b9096fe6-7e29-418d-97be-ded94d123693-kube-api-access-nlfkg\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:52:42.182165 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.182124 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:52:42.182312 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.182187 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 00:52:42.217039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.217008 4517 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:52:42.217187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.217064 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 00:52:42.754304 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.754261 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" event={"ID":"b9096fe6-7e29-418d-97be-ded94d123693","Type":"ContainerDied","Data":"fbb38bb853b30477abaa7cd0f45d2c6d5499c001d08f08c3957ee4ac64458073"} Oct 30 00:52:42.754511 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.754329 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbb38bb853b30477abaa7cd0f45d2c6d5499c001d08f08c3957ee4ac64458073" Oct 30 00:52:42.754511 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:42.754400 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph" Oct 30 00:52:43.186815 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:43.186782 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:43.187268 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:43.187229 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:43.188159 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:43.188138 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 00:52:43.220857 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:43.220834 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:43.221053 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:43.221019 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:43.221153 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:43.221142 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:43.221860 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:43.221843 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 00:52:44.943398 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:44.943367 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:46.484610 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.484569 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr"] Oct 30 00:52:46.485801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.485771 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b9096fe6-7e29-418d-97be-ded94d123693" containerName="revoke-keys" Oct 30 00:52:46.485801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.485796 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9096fe6-7e29-418d-97be-ded94d123693" containerName="revoke-keys" Oct 30 00:52:46.485979 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.485919 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9096fe6-7e29-418d-97be-ded94d123693" containerName="revoke-keys" Oct 30 00:52:46.489623 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.489602 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.519067 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.519038 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1e708b2f-serving-cert\"" Oct 30 00:52:46.526001 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.525966 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr"] Oct 30 00:52:46.526582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.526526 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q452\" (UniqueName: \"kubernetes.io/projected/b3680438-4918-4f52-be0f-fcdb4b098d2b-kube-api-access-2q452\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.527005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.526963 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.527282 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.527252 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3680438-4918-4f52-be0f-fcdb4b098d2b-openshift-service-ca\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.527372 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.527304 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-metrics-certs\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.580754 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.580702 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:46.628663 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.628627 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.628849 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.628804 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3680438-4918-4f52-be0f-fcdb4b098d2b-openshift-service-ca\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.628849 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.628842 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-metrics-certs\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.629028 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.629010 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q452\" (UniqueName: \"kubernetes.io/projected/b3680438-4918-4f52-be0f-fcdb4b098d2b-kube-api-access-2q452\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.629474 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:52:46.628809 4517 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-serving-cert: secret "maas-api-e2e-migrate-1e708b2f-serving-cert" not found Oct 30 00:52:46.629578 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.629487 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3680438-4918-4f52-be0f-fcdb4b098d2b-openshift-service-ca\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.629578 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:52:46.629529 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls podName:b3680438-4918-4f52-be0f-fcdb4b098d2b nodeName:}" failed. No retries permitted until 2026-10-30 00:52:47.129509152 +0000 UTC m=+1105.041636278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls") pod "maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" (UID: "b3680438-4918-4f52-be0f-fcdb4b098d2b") : secret "maas-api-e2e-migrate-1e708b2f-serving-cert" not found Oct 30 00:52:46.632590 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.632562 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-metrics-certs\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.651269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.651245 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q452\" (UniqueName: \"kubernetes.io/projected/b3680438-4918-4f52-be0f-fcdb4b098d2b-kube-api-access-2q452\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:46.695275 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.695237 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:46.910236 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:46.910199 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:47.135442 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:47.135410 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:47.138009 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:47.137981 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls\") pod \"maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:47.241098 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:47.241003 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:47.384238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:47.384199 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:47.405357 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:47.405327 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:47.612971 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:47.612937 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:47.771841 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:47.771817 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr"] Oct 30 00:52:47.806059 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:47.806011 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" event={"ID":"b3680438-4918-4f52-be0f-fcdb4b098d2b","Type":"ContainerStarted","Data":"f192c08305cc4db6fb5003796bcc0b14b9920eaa81955aa5e25e090192e3a40a"} Oct 30 00:52:48.626449 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:48.626410 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:48.822876 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:48.822836 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" event={"ID":"b3680438-4918-4f52-be0f-fcdb4b098d2b","Type":"ContainerStarted","Data":"1c8fe4fadfe7f8f9280207caeef74548642a127e0aae0bb4325da47dd3fadce0"} Oct 30 00:52:48.823942 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:48.823920 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:48.874381 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:48.874317 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" podStartSLOduration=2.590383817 podStartE2EDuration="2.874297192s" totalImagesPullingTime="283.913375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:48.868152928 +0000 UTC m=+1106.780280070" watchObservedRunningTime="2026-10-30 00:52:48.874297192 +0000 UTC m=+1106.786424336" Oct 30 00:52:49.282015 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:49.281956 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:50.339295 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.339266 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:50.380834 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.380804 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn"] Oct 30 00:52:50.390764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.390738 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.404624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.404597 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn"] Oct 30 00:52:50.410524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.409728 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-74b07396-serving-cert\"" Oct 30 00:52:50.483502 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.483461 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:50.497536 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.497510 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae64730-60a2-42c3-98e7-1f3bc9851609-openshift-service-ca\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.497680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.497592 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-maas-api-tls\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.497775 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.497759 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-metrics-certs\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.498003 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.497985 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fd6j\" (UniqueName: \"kubernetes.io/projected/1ae64730-60a2-42c3-98e7-1f3bc9851609-kube-api-access-4fd6j\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.602163 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.599931 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fd6j\" (UniqueName: \"kubernetes.io/projected/1ae64730-60a2-42c3-98e7-1f3bc9851609-kube-api-access-4fd6j\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.602163 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.600061 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae64730-60a2-42c3-98e7-1f3bc9851609-openshift-service-ca\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.602163 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.600129 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-maas-api-tls\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.602163 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.600252 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-metrics-certs\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.602931 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.602897 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:50.604118 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.604087 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae64730-60a2-42c3-98e7-1f3bc9851609-openshift-service-ca\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.604331 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.604312 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-metrics-certs\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.622043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.621303 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-maas-api-tls\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.637697 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.637104 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fd6j\" (UniqueName: \"kubernetes.io/projected/1ae64730-60a2-42c3-98e7-1f3bc9851609-kube-api-access-4fd6j\") pod \"maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:50.709227 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:50.709193 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:51.032246 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.032222 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn"] Oct 30 00:52:51.034054 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:52:51.034022 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae64730_60a2_42c3_98e7_1f3bc9851609.slice/crio-6221d7df342774707264670d4aba41766053260da37c42e29c8798fa41dee62c WatchSource:0}: Error finding container 6221d7df342774707264670d4aba41766053260da37c42e29c8798fa41dee62c: Status 404 returned error can't find the container with id 6221d7df342774707264670d4aba41766053260da37c42e29c8798fa41dee62c Oct 30 00:52:51.454217 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.452546 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5"] Oct 30 00:52:51.480110 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.480085 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.488039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.481479 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5"] Oct 30 00:52:51.495147 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.495123 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 00:52:51.497441 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.497421 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pqwzr\"" Oct 30 00:52:51.497557 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.497493 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 00:52:51.515453 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.515417 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 00:52:51.622539 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.622456 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csldg\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-kube-api-access-csldg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.622539 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.622490 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.622784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.622754 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.622849 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.622808 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.622849 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.622844 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.622993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.622900 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.622993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.622965 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94b14c6-f1c5-433f-9843-9c7aefea72c0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.726543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.724441 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.726543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.724501 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.726543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.724540 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.726543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.724611 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94b14c6-f1c5-433f-9843-9c7aefea72c0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.726543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.725788 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csldg\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-kube-api-access-csldg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.726543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.725851 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.726543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.725904 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.726543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.726054 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.730205 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.730138 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.731610 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.731585 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.732013 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.731992 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94b14c6-f1c5-433f-9843-9c7aefea72c0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.732091 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.732021 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.751467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.746585 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.751467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.747913 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csldg\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-kube-api-access-csldg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qbkz5\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.819128 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.819097 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:51.866294 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.865519 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" event={"ID":"1ae64730-60a2-42c3-98e7-1f3bc9851609","Type":"ContainerStarted","Data":"3aa6324d31670f7945193f13465866dad40928db6c06bef28ef2206f14c779d0"} Oct 30 00:52:51.866294 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.865597 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" event={"ID":"1ae64730-60a2-42c3-98e7-1f3bc9851609","Type":"ContainerStarted","Data":"6221d7df342774707264670d4aba41766053260da37c42e29c8798fa41dee62c"} Oct 30 00:52:51.866294 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.865845 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:51.931555 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:51.915428 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" podStartSLOduration=1.581095074 podStartE2EDuration="1.915408185s" totalImagesPullingTime="334.313111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:51.914417413 +0000 UTC m=+1109.826544577" watchObservedRunningTime="2026-10-30 00:52:51.915408185 +0000 UTC m=+1109.827535328" Oct 30 00:52:52.181735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:52.179028 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5"] Oct 30 00:52:52.399742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:52.399671 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:52.510531 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:52.510458 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:52.622486 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:52.622452 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:52.876128 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:52.876076 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" event={"ID":"a94b14c6-f1c5-433f-9843-9c7aefea72c0","Type":"ContainerStarted","Data":"9bf0eb7bd47b1e5fcf5e74039c15fcac309241ba1da66a814b373f1bca9f1605"} Oct 30 00:52:53.253971 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:53.253880 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:54.010295 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:54.010260 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:54.125903 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:54.125870 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:54.373265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:54.373179 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:54.594257 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:54.594163 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:55.761796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:55.761761 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:55.762188 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:55.761825 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:52:55.856597 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:55.856572 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:52:55.903481 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:55.903443 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" event={"ID":"a94b14c6-f1c5-433f-9843-9c7aefea72c0","Type":"ContainerStarted","Data":"0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc"} Oct 30 00:52:55.903681 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:55.903541 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:52:55.951983 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:55.951926 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" podStartSLOduration=1.377337472 podStartE2EDuration="4.95190726s" totalImagesPullingTime="3.574569788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:55.949852818 +0000 UTC m=+1113.861979990" watchObservedRunningTime="2026-10-30 00:52:55.95190726 +0000 UTC m=+1113.864034401" Oct 30 00:52:56.144410 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:56.144376 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:56.893757 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:56.893683 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:56.914551 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:56.914521 4517 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qbkz5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:52:56.914721 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:56.914580 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" podUID="a94b14c6-f1c5-433f-9843-9c7aefea72c0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:52:57.882727 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:57.882675 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:52:57.968890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:57.968856 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:59.238949 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:59.238916 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:59.574542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:59.574514 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:59.914208 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:52:59.914118 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 00:53:00.361822 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:00.359591 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:00.582183 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:00.582149 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:00.759267 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:53:00.759188 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:53:01.564520 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:01.564484 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:01.692909 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:01.692862 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:02.332272 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:02.332046 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:03.433896 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:03.432691 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:03.532049 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:03.531321 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:53:03.620004 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:03.619314 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:53:03.656086 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:03.656046 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:03.690779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:03.686812 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:03.690779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:03.687224 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:53:03.988797 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:03.970799 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.060729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.059926 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.263891 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.263789 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.390853 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.390807 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.501779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.485833 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.501779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.486437 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:53:04.505211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.505178 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.613810 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.610433 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.614295 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.614261 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:53:04.615177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.615155 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:53:04.662941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.662798 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.670763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.663270 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:53:04.795734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.758999 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:53:04.796170 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.796144 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:53:04.945784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.944972 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd"] Oct 30 00:53:04.989731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:04.989137 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.087733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.082811 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.146386 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.146344 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:53:05.227045 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.226935 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd"] Oct 30 00:53:05.252820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.224556 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.253119 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.175770 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.254120 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.254099 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.284390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.283579 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.284390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.283652 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tl5d\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-kube-api-access-2tl5d\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.284390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.283725 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.284390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.283832 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.284390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.284066 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.284390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.284110 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.284390 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.284179 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.532159 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.532207 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.532273 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.532318 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.532341 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.532411 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.532451 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tl5d\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-kube-api-access-2tl5d\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.532549 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.532663 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.534342 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.535037 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.536794 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.537316 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.542220 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.561297 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.562075 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.563233 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.601693 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.632833 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.637177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.633448 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tl5d\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-kube-api-access-2tl5d\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.640072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.640001 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.664974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.664931 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.676808 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.675782 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.719662 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.719560 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.787997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.787967 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:53:05.788278 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.788242 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.796624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.796595 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:53:05.880733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.855984 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:05.880733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.857819 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.999753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:05.993246 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:53:06.054008 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.053829 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:06.057769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.055133 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:53:06.057769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.056928 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:06.101759 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.094271 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:06.178735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.169486 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:53:06.237788 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.227357 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:53:06.638735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.638069 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:53:06.664134 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.641316 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:53:06.688486 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.688442 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:06.822735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.819434 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:53:06.923806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.922946 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:06.986733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:06.958598 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:53:07.104733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.080262 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:07.351632 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.351585 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:07.432129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.429593 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:07.657736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.644946 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:53:07.696740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.691850 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:07.737151 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.710191 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:53:07.737437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.737400 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:53:07.764850 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.764810 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:07.844405 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.844349 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:08.120805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:07.985806 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:53:08.240735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.088256 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:53:08.240735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.088400 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" podUID="23dedc80-2107-4d18-be9f-85405fcfa02c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:08.240735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.176273 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:53:08.260920 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.196604 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:53:08.261476 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.200037 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:08.339606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.331656 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:08.407964 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.407935 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:08.433140 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.433082 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:53:08.433484 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.433455 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:53:08.433747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.433696 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:53:08.541565 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.541534 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:53:08.543874 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.543849 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd"] Oct 30 00:53:08.672780 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.671973 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:53:08.709956 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.709920 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:53:08.776970 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.776911 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:53:08.777865 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.777844 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:53:08.815617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.815459 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:08.865733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.847574 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:08.865733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:08.848352 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:53:09.091195 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.090069 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:53:09.253840 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.238197 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.253840 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.239526 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.253840 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.239756 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.253840 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.240355 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:53:09.318355 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.317934 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.382897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.382387 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.388524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.387294 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:53:09.565448 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.565415 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" event={"ID":"9d6f3ac9-1b47-4fe0-a82e-165a947bd458","Type":"ContainerStarted","Data":"1bf761b54700146cedb307790d134565e84384cba661613f2d0b93eb45901d4c"} Oct 30 00:53:09.682904 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.682805 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.762585 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.738636 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.772739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.763955 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.807140 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.807101 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:53:09.824740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.807763 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 00:53:09.892850 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.892800 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.968526 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.968429 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:53:09.970156 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:09.970136 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:53:10.119733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.094826 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:10.121891 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.121864 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:53:10.336155 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.336106 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:53:10.346162 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.346119 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:10.348568 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.348528 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:10.427380 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.427339 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:10.427978 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.427951 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 00:53:10.429131 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.428352 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 00:53:10.455742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.454544 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:53:10.455742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.454639 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:53:10.455742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.454689 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:53:10.481382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.481354 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:53:10.494734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.487228 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:53:10.592569 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.592484 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" event={"ID":"9d6f3ac9-1b47-4fe0-a82e-165a947bd458","Type":"ContainerStarted","Data":"32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9"} Oct 30 00:53:10.690954 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.690022 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:53:10.702351 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.701500 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:53:10.763199 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.763113 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:10.790805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.789795 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" podStartSLOduration=6.789777256 podStartE2EDuration="6.789777256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:10.770645961 +0000 UTC m=+1128.682773127" watchObservedRunningTime="2026-10-30 00:53:10.789777256 +0000 UTC m=+1128.701904399" Oct 30 00:53:10.862340 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.862262 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:10.862640 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.862596 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:53:10.862899 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.862810 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 00:53:10.934730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:10.933983 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:53:11.020128 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:11.020083 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:53:11.280698 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:11.280600 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:11.861554 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:11.859044 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:53:11.861554 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:11.859102 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 00:53:12.866752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:12.866038 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:53:12.866752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:12.866098 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 00:53:13.292222 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:13.291237 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:13.865764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:13.862190 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:53:13.865764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:13.862243 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 00:53:13.896300 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:13.896220 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:14.014095 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:14.014065 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:14.417593 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:14.417560 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:14.880270 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:14.880234 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:53:14.880605 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:14.880558 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 00:53:15.385519 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:15.385329 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:15.870735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:15.869845 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:53:15.870735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:15.869907 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 00:53:16.841486 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:16.841455 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:16.858766 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:16.858695 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:53:16.858905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:16.858795 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 00:53:16.952413 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:16.952375 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:17.031678 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:17.031638 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn"] Oct 30 00:53:17.043244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:17.043212 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3eb6a08-2lzmn"] Oct 30 00:53:17.942513 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:17.942475 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:18.183298 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:18.183269 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:18.183575 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:18.183555 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:18.184493 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:18.184475 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 00:53:18.235826 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:18.235699 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b7c7cf-b72c-45b6-b297-f07e1ace4ef3" path="/var/lib/kubelet/pods/21b7c7cf-b72c-45b6-b297-f07e1ace4ef3/volumes" Oct 30 00:53:19.874256 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:19.874223 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:21.872060 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:21.872024 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:23.623177 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:23.623099 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:24.152000 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:24.151961 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:26.029812 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:26.029778 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:27.083020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:27.082987 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:27.206915 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:27.206873 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:28.437012 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:28.436979 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:29.028742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:29.028695 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp"] Oct 30 00:53:29.038238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:29.038212 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21946d9d-26pjp"] Oct 30 00:53:30.123095 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:30.123065 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:30.238534 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:30.238500 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc8759e-efb0-4a64-94ae-48ccf2a8c798" path="/var/lib/kubelet/pods/3fc8759e-efb0-4a64-94ae-48ccf2a8c798/volumes" Oct 30 00:53:30.759908 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:53:30.759867 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:53:31.862160 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:31.862129 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:33.373263 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:33.372958 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:33.492770 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:33.492691 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:33.920099 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:33.920061 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:36.497465 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:36.495806 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.583301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:36.568264 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.630515 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:36.630380 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:36.759862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:36.735779 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.818940 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:36.817900 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.818940 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:36.818183 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:36.859893 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:36.859843 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.892948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:36.892876 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.154034 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.153990 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:53:37.168317 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.168257 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.180725 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.180660 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.241027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.240976 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:37.306010 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.303925 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 00:53:37.399808 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.393879 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.432740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.432628 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:37.484186 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.482982 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.574493 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.573847 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.598734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.575263 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:53:37.753903 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.753808 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.755730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.755669 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.762090 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.758986 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:53:37.844836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.830876 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.844836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.832438 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:53:37.875937 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.875893 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:53:37.876897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:37.876872 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:53:38.069738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.068653 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:38.174784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.162333 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:53:38.175520 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.175491 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:53:38.221166 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.183657 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.221166 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.203277 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:53:38.249038 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.245306 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.388790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.376820 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:53:38.388790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.376887 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:53:38.543733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.489786 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.543733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.490289 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:53:38.543733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.515084 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 00:53:38.567734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.565798 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.567734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.566278 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:53:38.627733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.624304 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:53:38.688031 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.687845 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.841342 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.841298 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.964064 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.963966 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.966835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.966791 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:39.035926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.967053 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:39.035926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:38.976940 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:53:39.035926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.030078 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:53:39.059024 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.051941 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:39.223807 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.218084 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:53:39.249988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.231025 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:39.399525 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.399470 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:39.490401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.467754 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:53:39.490401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.473590 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:53:39.548733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.538033 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:39.548733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.538663 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:53:39.589967 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.589920 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:39.777793 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.738487 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:53:39.865316 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.865274 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:53:39.989035 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:39.984849 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.003209 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.003163 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:53:40.028810 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.014830 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:53:40.034746 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.030862 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:53:40.149738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.036281 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:53:40.403063 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.403025 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.476657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.405656 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:40.477228 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.435780 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.477990 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.444185 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.503735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.468901 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 00:53:40.504383 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.475820 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.559130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.559081 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.619740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.617090 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:40.619740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.619576 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:53:40.619740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.619630 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:53:40.698904 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.698819 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:53:40.739104 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.739066 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.795871 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.795806 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.845320 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.845293 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:53:40.866893 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.866843 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:53:40.885431 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.885400 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:53:40.887892 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.887859 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:40.906260 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.904188 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:53:40.906499 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.904224 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded" Oct 30 00:53:40.924833 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.923070 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr"] Oct 30 00:53:40.924833 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.923297 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" podUID="b3680438-4918-4f52-be0f-fcdb4b098d2b" containerName="maas-api" containerID="cri-o://1c8fe4fadfe7f8f9280207caeef74548642a127e0aae0bb4325da47dd3fadce0" gracePeriod=30 Oct 30 00:53:40.925978 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:40.925959 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:53:41.075213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.026342 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:41.075213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.029475 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:41.075213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.040854 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:41.075213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.049465 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:41.075213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.049987 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:53:41.075213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.051614 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 00:53:41.075213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.053162 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:41.246835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.238819 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:53:41.483446 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.454551 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:53:41.513662 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.508052 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:41.513662 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.508381 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 00:53:41.513662 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.508649 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:53:41.538769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.535452 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:53:41.563904 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.559785 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:53:41.811790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.800377 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:41.931731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:41.912557 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:42.032870 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.032826 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:42.071806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.065215 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:42.343826 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.278050 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Oct 30 00:53:42.448862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.389771 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:42.556920 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.556870 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:42.559693 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.559657 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:42.581278 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.581237 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:42.596799 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.585054 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 00:53:42.720749 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.707171 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:42.789827 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:42.788830 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 00:53:43.150130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.106848 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:53:43.150130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.122687 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:43.150130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.131535 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 00:53:43.196194 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.186401 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 00:53:43.281116 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.280921 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:43.289096 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.287998 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:43.306347 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.306304 4517 generic.go:356] "Generic (PLEG): container finished" podID="b3680438-4918-4f52-be0f-fcdb4b098d2b" containerID="1c8fe4fadfe7f8f9280207caeef74548642a127e0aae0bb4325da47dd3fadce0" exitCode=0 Oct 30 00:53:43.379754 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.331669 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" event={"ID":"b3680438-4918-4f52-be0f-fcdb4b098d2b","Type":"ContainerDied","Data":"1c8fe4fadfe7f8f9280207caeef74548642a127e0aae0bb4325da47dd3fadce0"} Oct 30 00:53:43.420758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.413897 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:53:43.467731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.450278 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:53:43.481913 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.470506 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3680438-4918-4f52-be0f-fcdb4b098d2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3680438-4918-4f52-be0f-fcdb4b098d2b-openshift-service-ca\") pod \"b3680438-4918-4f52-be0f-fcdb4b098d2b\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " Oct 30 00:53:43.486939 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.486909 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3680438-4918-4f52-be0f-fcdb4b098d2b-kube-api-access-2q452\" (UniqueName: \"kubernetes.io/projected/b3680438-4918-4f52-be0f-fcdb4b098d2b-kube-api-access-2q452\") pod \"b3680438-4918-4f52-be0f-fcdb4b098d2b\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " Oct 30 00:53:43.487772 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.487749 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-metrics-certs\") pod \"b3680438-4918-4f52-be0f-fcdb4b098d2b\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " Oct 30 00:53:43.495286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.495263 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls\") pod \"b3680438-4918-4f52-be0f-fcdb4b098d2b\" (UID: \"b3680438-4918-4f52-be0f-fcdb4b098d2b\") " Oct 30 00:53:43.505317 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.505290 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3680438-4918-4f52-be0f-fcdb4b098d2b-openshift-service-ca" pod "b3680438-4918-4f52-be0f-fcdb4b098d2b" (UID: "b3680438-4918-4f52-be0f-fcdb4b098d2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:43.526653 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.526620 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3680438-4918-4f52-be0f-fcdb4b098d2b-kube-api-access-2q452" pod "b3680438-4918-4f52-be0f-fcdb4b098d2b" (UID: "b3680438-4918-4f52-be0f-fcdb4b098d2b"). InnerVolumeSpecName "kube-api-access-2q452". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:43.530432 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.530400 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-metrics-certs" pod "b3680438-4918-4f52-be0f-fcdb4b098d2b" (UID: "b3680438-4918-4f52-be0f-fcdb4b098d2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:43.530685 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.530667 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls" pod "b3680438-4918-4f52-be0f-fcdb4b098d2b" (UID: "b3680438-4918-4f52-be0f-fcdb4b098d2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:43.559328 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.559297 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:53:43.608744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.588805 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:43.608744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.590644 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 00:53:43.629983 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.628449 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3680438-4918-4f52-be0f-fcdb4b098d2b-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:43.629983 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.628480 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q452\" (UniqueName: \"kubernetes.io/projected/b3680438-4918-4f52-be0f-fcdb4b098d2b-kube-api-access-2q452\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:43.629983 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.628495 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:43.629983 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.628508 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3680438-4918-4f52-be0f-fcdb4b098d2b-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:43.637607 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.637441 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:43.680723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.679802 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 00:53:43.687796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.687767 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn"] Oct 30 00:53:43.688321 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.688292 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" podUID="1ae64730-60a2-42c3-98e7-1f3bc9851609" containerName="maas-api" containerID="cri-o://3aa6324d31670f7945193f13465866dad40928db6c06bef28ef2206f14c779d0" gracePeriod=30 Oct 30 00:53:43.728043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.727999 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:43.833986 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.833892 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:43.835097 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.835074 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:53:43.868526 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:43.863525 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:44.027985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.027951 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:44.140096 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.140006 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:53:44.194773 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.187819 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.226847 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.226801 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.227471 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.227447 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:53:44.319753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.294774 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 30 00:53:44.319753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.292808 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 30 00:53:44.319753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.301296 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.319753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.314722 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.413437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.413088 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:44.504749 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.474845 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.512732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.508326 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.570741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.566950 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.609982 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.609869 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.612255 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.611360 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 00:53:44.613977 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.613956 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:53:44.701814 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.615660 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:44.702112 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.667220 4517 generic.go:356] "Generic (PLEG): container finished" podID="1ae64730-60a2-42c3-98e7-1f3bc9851609" containerID="3aa6324d31670f7945193f13465866dad40928db6c06bef28ef2206f14c779d0" exitCode=0 Oct 30 00:53:44.702415 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.680101 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.703733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.702398 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:53:44.794030 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.720149 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" event={"ID":"1ae64730-60a2-42c3-98e7-1f3bc9851609","Type":"ContainerDied","Data":"3aa6324d31670f7945193f13465866dad40928db6c06bef28ef2206f14c779d0"} Oct 30 00:53:44.794030 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.720314 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:53:44.794030 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.739272 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.882860 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.882826 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm"] Oct 30 00:53:44.899732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.883277 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" podUID="84bd9741-2ea2-4e2d-bdb0-25c486aab26f" containerName="istio-proxy" containerID="cri-o://77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75" gracePeriod=30 Oct 30 00:53:44.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.897385 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:44.988403 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:44.897434 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:45.004763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.004733 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" Oct 30 00:53:45.009204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.005813 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr" event={"ID":"b3680438-4918-4f52-be0f-fcdb4b098d2b","Type":"ContainerDied","Data":"f192c08305cc4db6fb5003796bcc0b14b9920eaa81955aa5e25e090192e3a40a"} Oct 30 00:53:45.009204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.007139 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 00:53:45.009204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.007871 4517 scope.go:122] "RemoveContainer" containerID="1c8fe4fadfe7f8f9280207caeef74548642a127e0aae0bb4325da47dd3fadce0" Oct 30 00:53:45.009204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.008543 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:45.130762 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.103270 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 00:53:45.155001 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.149536 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 00:53:45.195346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.194100 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:45.241818 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.229542 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 00:53:45.367366 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.278649 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 00:53:45.367827 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.311184 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:45.368062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.315809 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:45.488825 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.488793 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4"] Oct 30 00:53:45.489279 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.489246 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" podUID="8b4a30d0-d849-41cf-be6c-de15b996862a" containerName="istio-proxy" containerID="cri-o://b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a" gracePeriod=30 Oct 30 00:53:45.508151 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.503376 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:45.550733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.546641 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 00:53:45.581771 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.572272 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:45.585908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.585867 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:53:45.637756 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.611813 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:45.734764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.708416 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded" Oct 30 00:53:46.013771 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:45.992869 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:46.014234 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.014190 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:46.052735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.016171 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:53:46.091734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.024200 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:46.151792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.105062 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:46.151792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.119213 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 00:53:46.167809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.152342 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:46.234734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.155749 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:46.396732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.395771 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 00:53:46.440445 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.438246 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:46.447922 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.447875 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:46.626744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.625894 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:53:46.634733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.627998 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 00:53:46.634733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.628050 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 00:53:46.634733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.628076 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 00:53:46.634733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.628099 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr"] Oct 30 00:53:46.634733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.628118 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e708b2f-597d998d9d-jpcgr"] Oct 30 00:53:46.634733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.628154 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 00:53:46.634733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.628189 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 00:53:46.634733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.628212 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 00:53:46.634733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.633844 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 00:53:46.637044 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.637019 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:46.637659 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.637630 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" event={"ID":"1ae64730-60a2-42c3-98e7-1f3bc9851609","Type":"ContainerDied","Data":"6221d7df342774707264670d4aba41766053260da37c42e29c8798fa41dee62c"} Oct 30 00:53:46.638043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.637971 4517 scope.go:122] "RemoveContainer" containerID="3aa6324d31670f7945193f13465866dad40928db6c06bef28ef2206f14c779d0" Oct 30 00:53:46.815269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.815222 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:46.839184 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.839143 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-metrics-certs\") pod \"1ae64730-60a2-42c3-98e7-1f3bc9851609\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " Oct 30 00:53:46.841620 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.841585 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae64730-60a2-42c3-98e7-1f3bc9851609-kube-api-access-4fd6j\" (UniqueName: \"kubernetes.io/projected/1ae64730-60a2-42c3-98e7-1f3bc9851609-kube-api-access-4fd6j\") pod \"1ae64730-60a2-42c3-98e7-1f3bc9851609\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " Oct 30 00:53:46.841797 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.841683 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-maas-api-tls\") pod \"1ae64730-60a2-42c3-98e7-1f3bc9851609\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " Oct 30 00:53:46.841797 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.841763 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae64730-60a2-42c3-98e7-1f3bc9851609-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae64730-60a2-42c3-98e7-1f3bc9851609-openshift-service-ca\") pod \"1ae64730-60a2-42c3-98e7-1f3bc9851609\" (UID: \"1ae64730-60a2-42c3-98e7-1f3bc9851609\") " Oct 30 00:53:46.842923 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.842889 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae64730-60a2-42c3-98e7-1f3bc9851609-openshift-service-ca" pod "1ae64730-60a2-42c3-98e7-1f3bc9851609" (UID: "1ae64730-60a2-42c3-98e7-1f3bc9851609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:46.845352 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.845330 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-metrics-certs" pod "1ae64730-60a2-42c3-98e7-1f3bc9851609" (UID: "1ae64730-60a2-42c3-98e7-1f3bc9851609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:46.845500 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.845481 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-maas-api-tls" pod "1ae64730-60a2-42c3-98e7-1f3bc9851609" (UID: "1ae64730-60a2-42c3-98e7-1f3bc9851609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:46.845749 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:46.845731 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae64730-60a2-42c3-98e7-1f3bc9851609-kube-api-access-4fd6j" pod "1ae64730-60a2-42c3-98e7-1f3bc9851609" (UID: "1ae64730-60a2-42c3-98e7-1f3bc9851609"). InnerVolumeSpecName "kube-api-access-4fd6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:47.009894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:47.001344 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:47.009894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:47.001381 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fd6j\" (UniqueName: \"kubernetes.io/projected/1ae64730-60a2-42c3-98e7-1f3bc9851609-kube-api-access-4fd6j\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:47.009894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:47.001398 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae64730-60a2-42c3-98e7-1f3bc9851609-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:47.009894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:47.002814 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae64730-60a2-42c3-98e7-1f3bc9851609-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:47.298169 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:47.298088 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:47.661843 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:47.661135 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn" Oct 30 00:53:47.729000 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:47.727643 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn"] Oct 30 00:53:47.752612 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:47.752314 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74b07396-6777ff95b7-8dcxn"] Oct 30 00:53:48.245037 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:48.244999 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae64730-60a2-42c3-98e7-1f3bc9851609" path="/var/lib/kubelet/pods/1ae64730-60a2-42c3-98e7-1f3bc9851609/volumes" Oct 30 00:53:48.245608 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:48.245586 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3680438-4918-4f52-be0f-fcdb4b098d2b" path="/var/lib/kubelet/pods/b3680438-4918-4f52-be0f-fcdb4b098d2b/volumes" Oct 30 00:53:48.515614 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:48.515524 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:49.225818 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:49.225763 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:49.349274 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:49.349239 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:50.560793 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:50.560753 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:50.944871 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:50.944780 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:51.065735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:51.065617 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:51.993771 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:51.993030 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:53:52.050475 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.050456 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:53:52.050966 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.050942 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-credential-socket\") pod \"8b4a30d0-d849-41cf-be6c-de15b996862a\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " Oct 30 00:53:52.051045 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.050989 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-envoy\") pod \"8b4a30d0-d849-41cf-be6c-de15b996862a\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " Oct 30 00:53:52.051045 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.051041 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-token\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-token\") pod \"8b4a30d0-d849-41cf-be6c-de15b996862a\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " Oct 30 00:53:52.051164 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.051067 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-kube-api-access-knbh4\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-kube-api-access-knbh4\") pod \"8b4a30d0-d849-41cf-be6c-de15b996862a\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " Oct 30 00:53:52.051164 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.051141 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-podinfo\") pod \"8b4a30d0-d849-41cf-be6c-de15b996862a\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " Oct 30 00:53:52.051266 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.051232 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-credential-socket" pod "8b4a30d0-d849-41cf-be6c-de15b996862a" (UID: "8b4a30d0-d849-41cf-be6c-de15b996862a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.051348 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.051330 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4a30d0-d849-41cf-be6c-de15b996862a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4a30d0-d849-41cf-be6c-de15b996862a-istiod-ca-cert\") pod \"8b4a30d0-d849-41cf-be6c-de15b996862a\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " Oct 30 00:53:52.051421 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.051382 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-data\") pod \"8b4a30d0-d849-41cf-be6c-de15b996862a\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " Oct 30 00:53:52.051981 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.051565 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-socket\") pod \"8b4a30d0-d849-41cf-be6c-de15b996862a\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " Oct 30 00:53:52.051981 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.051595 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-certs\") pod \"8b4a30d0-d849-41cf-be6c-de15b996862a\" (UID: \"8b4a30d0-d849-41cf-be6c-de15b996862a\") " Oct 30 00:53:52.051981 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.051868 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4a30d0-d849-41cf-be6c-de15b996862a-istiod-ca-cert" pod "8b4a30d0-d849-41cf-be6c-de15b996862a" (UID: "8b4a30d0-d849-41cf-be6c-de15b996862a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.052050 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-socket" pod "8b4a30d0-d849-41cf-be6c-de15b996862a" (UID: "8b4a30d0-d849-41cf-be6c-de15b996862a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.052445 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-certs" pod "8b4a30d0-d849-41cf-be6c-de15b996862a" (UID: "8b4a30d0-d849-41cf-be6c-de15b996862a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.052525 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-data" pod "8b4a30d0-d849-41cf-be6c-de15b996862a" (UID: "8b4a30d0-d849-41cf-be6c-de15b996862a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.053409 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b4a30d0-d849-41cf-be6c-de15b996862a-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.053428 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.053439 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.053452 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.053467 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.053779 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-podinfo" pod "8b4a30d0-d849-41cf-be6c-de15b996862a" (UID: "8b4a30d0-d849-41cf-be6c-de15b996862a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.054086 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-token" pod "8b4a30d0-d849-41cf-be6c-de15b996862a" (UID: "8b4a30d0-d849-41cf-be6c-de15b996862a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:52.054805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.054478 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-envoy" pod "8b4a30d0-d849-41cf-be6c-de15b996862a" (UID: "8b4a30d0-d849-41cf-be6c-de15b996862a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.055764 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.055741 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-kube-api-access-knbh4" pod "8b4a30d0-d849-41cf-be6c-de15b996862a" (UID: "8b4a30d0-d849-41cf-be6c-de15b996862a"). InnerVolumeSpecName "kube-api-access-knbh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:52.154498 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.154464 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-envoy\") pod \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " Oct 30 00:53:52.154671 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.154560 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-kube-api-access-l495p\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-kube-api-access-l495p\") pod \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " Oct 30 00:53:52.154671 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.154589 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istiod-ca-cert\") pod \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " Oct 30 00:53:52.154671 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.154612 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-certs\") pod \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " Oct 30 00:53:52.154671 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.154635 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-credential-socket\") pod \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " Oct 30 00:53:52.154916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.154682 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-podinfo\") pod \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " Oct 30 00:53:52.154916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.154734 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-token\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-token\") pod \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " Oct 30 00:53:52.154916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.154775 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-data\") pod \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " Oct 30 00:53:52.154916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.154846 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-socket\") pod \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\" (UID: \"84bd9741-2ea2-4e2d-bdb0-25c486aab26f\") " Oct 30 00:53:52.155130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155103 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istiod-ca-cert" pod "84bd9741-2ea2-4e2d-bdb0-25c486aab26f" (UID: "84bd9741-2ea2-4e2d-bdb0-25c486aab26f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:52.155456 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155386 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-certs" pod "84bd9741-2ea2-4e2d-bdb0-25c486aab26f" (UID: "84bd9741-2ea2-4e2d-bdb0-25c486aab26f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.155456 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155396 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-data" pod "84bd9741-2ea2-4e2d-bdb0-25c486aab26f" (UID: "84bd9741-2ea2-4e2d-bdb0-25c486aab26f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.155456 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155442 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-credential-socket" pod "84bd9741-2ea2-4e2d-bdb0-25c486aab26f" (UID: "84bd9741-2ea2-4e2d-bdb0-25c486aab26f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.155725 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155573 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-socket" pod "84bd9741-2ea2-4e2d-bdb0-25c486aab26f" (UID: "84bd9741-2ea2-4e2d-bdb0-25c486aab26f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.155848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155830 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.155910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155849 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.155910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155857 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.155910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155865 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knbh4\" (UniqueName: \"kubernetes.io/projected/8b4a30d0-d849-41cf-be6c-de15b996862a-kube-api-access-knbh4\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.155910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155877 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.155910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155887 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.155910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155895 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b4a30d0-d849-41cf-be6c-de15b996862a-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.155910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155903 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.155910 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.155910 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.156993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.156968 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-kube-api-access-l495p" pod "84bd9741-2ea2-4e2d-bdb0-25c486aab26f" (UID: "84bd9741-2ea2-4e2d-bdb0-25c486aab26f"). InnerVolumeSpecName "kube-api-access-l495p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:52.157123 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.157106 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-token" pod "84bd9741-2ea2-4e2d-bdb0-25c486aab26f" (UID: "84bd9741-2ea2-4e2d-bdb0-25c486aab26f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:52.157302 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.157283 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-envoy" pod "84bd9741-2ea2-4e2d-bdb0-25c486aab26f" (UID: "84bd9741-2ea2-4e2d-bdb0-25c486aab26f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:52.157372 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.157355 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-podinfo" pod "84bd9741-2ea2-4e2d-bdb0-25c486aab26f" (UID: "84bd9741-2ea2-4e2d-bdb0-25c486aab26f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:53:52.257489 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.257459 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l495p\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-kube-api-access-l495p\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.257603 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.257493 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.257603 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.257506 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.257603 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.257517 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84bd9741-2ea2-4e2d-bdb0-25c486aab26f-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:53:52.407945 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.407908 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:52.718309 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.718271 4517 generic.go:356] "Generic (PLEG): container finished" podID="84bd9741-2ea2-4e2d-bdb0-25c486aab26f" containerID="77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75" exitCode=0 Oct 30 00:53:52.718492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.718346 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" Oct 30 00:53:52.718492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.718358 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" event={"ID":"84bd9741-2ea2-4e2d-bdb0-25c486aab26f","Type":"ContainerDied","Data":"77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75"} Oct 30 00:53:52.718492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.718418 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm" event={"ID":"84bd9741-2ea2-4e2d-bdb0-25c486aab26f","Type":"ContainerDied","Data":"d4e7c1f0f668d0eed4deb2768a8ec90f89c67747efddcb6918ccca86ac801240"} Oct 30 00:53:52.718492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.718448 4517 scope.go:122] "RemoveContainer" containerID="77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75" Oct 30 00:53:52.719763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.719742 4517 generic.go:356] "Generic (PLEG): container finished" podID="8b4a30d0-d849-41cf-be6c-de15b996862a" containerID="b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a" exitCode=0 Oct 30 00:53:52.719875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.719859 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" event={"ID":"8b4a30d0-d849-41cf-be6c-de15b996862a","Type":"ContainerDied","Data":"b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a"} Oct 30 00:53:52.719924 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.719892 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" event={"ID":"8b4a30d0-d849-41cf-be6c-de15b996862a","Type":"ContainerDied","Data":"d52c2749848c56c2268ac26f95e8507f6325b143a3a5373f610331f05d021d57"} Oct 30 00:53:52.719977 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.719967 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4" Oct 30 00:53:52.728207 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.728191 4517 scope.go:122] "RemoveContainer" containerID="77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75" Oct 30 00:53:52.728460 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:53:52.728436 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75\": container with ID starting with 77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75 not found: ID does not exist" containerID="77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75" Oct 30 00:53:52.728571 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.728465 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75"} err="failed to get container status \"77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75\": rpc error: code = NotFound desc = could not find container \"77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75\": container with ID starting with 77064876d99239bea406ec0d820f3446ff8b06a6b0da038d10c78024469e5a75 not found: ID does not exist" Oct 30 00:53:52.728571 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.728482 4517 scope.go:122] "RemoveContainer" containerID="b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a" Oct 30 00:53:52.737712 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.737691 4517 scope.go:122] "RemoveContainer" containerID="b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a" Oct 30 00:53:52.737974 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:53:52.737956 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a\": container with ID starting with b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a not found: ID does not exist" containerID="b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a" Oct 30 00:53:52.738034 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.737979 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a"} err="failed to get container status \"b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a\": rpc error: code = NotFound desc = could not find container \"b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a\": container with ID starting with b26d0b68d4a62d0a1c5c5e8ab6e0fb33a82c6b76cc3367677fbe0a7d06025c3a not found: ID does not exist" Oct 30 00:53:52.781699 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.781658 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4"] Oct 30 00:53:52.792961 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.792931 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c4e6c558-openshift-default-7977ff48f9-kxrv4"] Oct 30 00:53:52.816805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.816774 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm"] Oct 30 00:53:52.820763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:52.820737 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-74b07396-openshift-default-58f4bbdb7c-jswjm"] Oct 30 00:53:54.193671 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:54.193637 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:54.241169 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:54.241138 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84bd9741-2ea2-4e2d-bdb0-25c486aab26f" path="/var/lib/kubelet/pods/84bd9741-2ea2-4e2d-bdb0-25c486aab26f/volumes" Oct 30 00:53:54.241636 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:54.241622 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4a30d0-d849-41cf-be6c-de15b996862a" path="/var/lib/kubelet/pods/8b4a30d0-d849-41cf-be6c-de15b996862a/volumes" Oct 30 00:53:55.913685 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:55.913646 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:57.829223 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:57.829195 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:58.069217 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:58.069179 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:59.642583 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:53:59.642545 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:00.759191 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:54:00.759150 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:54:01.449045 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:01.449012 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:03.188171 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:03.187659 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:04.471495 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:04.471461 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:04.933819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:04.933784 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.148175 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.148144 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.267093 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.266985 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.387265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.387229 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.507318 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.507280 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.846426 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.846394 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v"] Oct 30 00:54:05.847042 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847022 4517 cpu_manager.go:414] "containerMap: removing container" podUID="8b4a30d0-d849-41cf-be6c-de15b996862a" containerName="istio-proxy" Oct 30 00:54:05.847042 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847042 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4a30d0-d849-41cf-be6c-de15b996862a" containerName="istio-proxy" Oct 30 00:54:05.847218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847094 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b3680438-4918-4f52-be0f-fcdb4b098d2b" containerName="maas-api" Oct 30 00:54:05.847218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847102 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3680438-4918-4f52-be0f-fcdb4b098d2b" containerName="maas-api" Oct 30 00:54:05.847218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847118 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1ae64730-60a2-42c3-98e7-1f3bc9851609" containerName="maas-api" Oct 30 00:54:05.847218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847125 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae64730-60a2-42c3-98e7-1f3bc9851609" containerName="maas-api" Oct 30 00:54:05.847218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847139 4517 cpu_manager.go:414] "containerMap: removing container" podUID="84bd9741-2ea2-4e2d-bdb0-25c486aab26f" containerName="istio-proxy" Oct 30 00:54:05.847218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847146 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bd9741-2ea2-4e2d-bdb0-25c486aab26f" containerName="istio-proxy" Oct 30 00:54:05.847820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847294 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3680438-4918-4f52-be0f-fcdb4b098d2b" containerName="maas-api" Oct 30 00:54:05.847820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847307 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae64730-60a2-42c3-98e7-1f3bc9851609" containerName="maas-api" Oct 30 00:54:05.847820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847318 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="84bd9741-2ea2-4e2d-bdb0-25c486aab26f" containerName="istio-proxy" Oct 30 00:54:05.847820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.847331 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4a30d0-d849-41cf-be6c-de15b996862a" containerName="istio-proxy" Oct 30 00:54:05.852857 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.852829 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.894826 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.892004 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v"] Oct 30 00:54:05.897652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.897154 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.897652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.897207 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e1387e-0825-4d7d-b5c5-e1282d033a11-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.897652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.897243 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.897652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.897381 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.898473 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.897425 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.898517 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.898638 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.898613 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-kube-api-access-9m569\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.898816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.898661 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.902897 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.899136 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:05.906652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.906624 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.940739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:05.940692 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-60e8f4-openshift-default-dockercfg-665t8\"" Oct 30 00:54:06.039785 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.029582 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.067907 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.060201 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.068217 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.068194 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.075844 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.068658 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.076062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.075799 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e1387e-0825-4d7d-b5c5-e1282d033a11-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.082844 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.076845 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e1387e-0825-4d7d-b5c5-e1282d033a11-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.083098 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.082800 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.083369 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.083349 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.083637 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.083497 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.090519 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.090484 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.090811 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.090783 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-kube-api-access-9m569\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.098362 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.098290 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.103768 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.099081 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.110735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.091984 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.112339 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.112310 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.140428 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.140398 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-kube-api-access-9m569\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.153003 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.152972 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.167436 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.167363 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.215613 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.215584 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:06.256596 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.256561 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:06.723384 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.723354 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v"] Oct 30 00:54:06.725052 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:54:06.725019 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e1387e_0825_4d7d_b5c5_e1282d033a11.slice/crio-06cf1737591ff90d86b286af8ac46a3e5544480db3bf109acd34cd851ebeece2 WatchSource:0}: Error finding container 06cf1737591ff90d86b286af8ac46a3e5544480db3bf109acd34cd851ebeece2: Status 404 returned error can't find the container with id 06cf1737591ff90d86b286af8ac46a3e5544480db3bf109acd34cd851ebeece2 Oct 30 00:54:06.728915 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.728774 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:54:06.729153 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.729121 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:54:06.729305 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.729289 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:54:06.823855 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:06.823823 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" event={"ID":"35e1387e-0825-4d7d-b5c5-e1282d033a11","Type":"ContainerStarted","Data":"06cf1737591ff90d86b286af8ac46a3e5544480db3bf109acd34cd851ebeece2"} Oct 30 00:54:07.058600 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:07.058541 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:07.833952 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:07.833918 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" event={"ID":"35e1387e-0825-4d7d-b5c5-e1282d033a11","Type":"ContainerStarted","Data":"d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621"} Oct 30 00:54:07.914734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:07.914458 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" podStartSLOduration=2.914435736 podStartE2EDuration="2.914435736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:07.885300046 +0000 UTC m=+1185.797427186" watchObservedRunningTime="2026-10-30 00:54:07.914435736 +0000 UTC m=+1185.826562876" Oct 30 00:54:08.217043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:08.216954 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:08.222740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:08.222694 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:08.738880 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:08.738762 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:08.847103 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:08.847071 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:08.848204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:08.848181 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:10.294521 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:10.294476 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:10.410513 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:10.410476 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:10.764890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:10.764790 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:11.865651 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:11.865618 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:11.987372 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:11.987330 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:12.109470 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:12.109444 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:12.953508 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:12.953478 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:13.698299 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:13.698267 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:14.532018 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:14.531981 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:14.665618 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:14.665581 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:16.253355 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:16.253299 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:18.135868 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:18.135833 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:19.821629 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:19.821594 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:20.143801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:20.143678 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:21.598196 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:21.598160 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:21.710260 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:21.710229 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:22.156877 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:22.156841 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:22.308554 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:22.308515 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:23.157420 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:23.157387 4517 scope.go:122] "RemoveContainer" containerID="b3afc39d7d83659e2867d60790152f0d0a27ee0fb30514046a63e391972b6360" Oct 30 00:54:23.596025 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:23.595974 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:25.698231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:25.698179 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:26.763131 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:26.763091 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:26.883229 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:26.883189 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:27.030411 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:27.030327 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5"] Oct 30 00:54:27.040436 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:27.040408 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ace35f-pb2h5"] Oct 30 00:54:27.934884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:27.934849 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:28.236083 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.236004 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e83e291a-a8de-44e2-91a9-3cd4824f6a97" path="/var/lib/kubelet/pods/e83e291a-a8de-44e2-91a9-3cd4824f6a97/volumes" Oct 30 00:54:28.508233 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.508150 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:28.620054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.620020 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:28.867091 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.862760 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5"] Oct 30 00:54:28.875733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.870155 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:28.875733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.871122 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5"] Oct 30 00:54:28.906839 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.906809 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d8090513\"" Oct 30 00:54:28.928400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.912301 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d8090513-dockercfg-5n5zl\"" Oct 30 00:54:28.992753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.992683 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8q5\" (UniqueName: \"kubernetes.io/projected/355623bc-0b4e-46e2-9077-6ea574101991-kube-api-access-sx8q5\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5\" (UID: \"355623bc-0b4e-46e2-9077-6ea574101991\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:28.993171 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:28.992910 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355623bc-0b4e-46e2-9077-6ea574101991-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5\" (UID: \"355623bc-0b4e-46e2-9077-6ea574101991\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:29.094860 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:29.094830 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355623bc-0b4e-46e2-9077-6ea574101991-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5\" (UID: \"355623bc-0b4e-46e2-9077-6ea574101991\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:29.095085 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:29.095047 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8q5\" (UniqueName: \"kubernetes.io/projected/355623bc-0b4e-46e2-9077-6ea574101991-kube-api-access-sx8q5\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5\" (UID: \"355623bc-0b4e-46e2-9077-6ea574101991\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:29.095582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:29.095554 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355623bc-0b4e-46e2-9077-6ea574101991-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5\" (UID: \"355623bc-0b4e-46e2-9077-6ea574101991\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:29.117586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:29.117504 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8q5\" (UniqueName: \"kubernetes.io/projected/355623bc-0b4e-46e2-9077-6ea574101991-kube-api-access-sx8q5\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5\" (UID: \"355623bc-0b4e-46e2-9077-6ea574101991\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:29.253025 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:29.252997 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:29.564689 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:29.564660 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5"] Oct 30 00:54:29.566513 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:54:29.566483 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355623bc_0b4e_46e2_9077_6ea574101991.slice/crio-f652a2dcfce7272740118546e692a843ee95a8fa7070368b17807813a2ae95c9 WatchSource:0}: Error finding container f652a2dcfce7272740118546e692a843ee95a8fa7070368b17807813a2ae95c9: Status 404 returned error can't find the container with id f652a2dcfce7272740118546e692a843ee95a8fa7070368b17807813a2ae95c9 Oct 30 00:54:30.029147 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:30.029048 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" event={"ID":"355623bc-0b4e-46e2-9077-6ea574101991","Type":"ContainerStarted","Data":"f652a2dcfce7272740118546e692a843ee95a8fa7070368b17807813a2ae95c9"} Oct 30 00:54:30.072729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:30.060090 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:30.174997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:30.174965 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:30.775098 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:54:30.775055 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:54:32.207156 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:32.207120 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:33.069574 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:33.069526 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" event={"ID":"355623bc-0b4e-46e2-9077-6ea574101991","Type":"ContainerStarted","Data":"66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c"} Oct 30 00:54:33.069939 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:33.069915 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:33.109552 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:33.109498 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" podStartSLOduration=2.709018735 podStartE2EDuration="5.109484272s" totalImagesPullingTime="2.400465537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:33.105454679 +0000 UTC m=+1211.017581820" watchObservedRunningTime="2026-10-30 00:54:33.109484272 +0000 UTC m=+1211.021611410" Oct 30 00:54:34.045230 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:34.044492 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:34.182131 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:34.182099 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:35.633302 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:35.633273 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:37.521316 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:37.521279 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:39.147914 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:39.147882 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:40.028591 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:40.028559 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t"] Oct 30 00:54:40.038972 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:40.038933 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcc7d03-jkp5t"] Oct 30 00:54:40.235468 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:40.235437 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c62e9b9-b4f1-4560-8179-26dbfa1120d5" path="/var/lib/kubelet/pods/0c62e9b9-b4f1-4560-8179-26dbfa1120d5/volumes" Oct 30 00:54:40.329851 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:40.329820 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:40.851901 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:40.851863 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:41.616090 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:41.616061 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:41.768142 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:41.768111 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:43.029219 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:43.029180 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:44.083574 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:44.083547 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:54:44.638317 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:44.638284 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:45.408023 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:45.407988 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:45.695418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:45.695324 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:45.828527 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:45.828494 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:45.956036 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:45.955949 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:46.088733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:46.074512 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:47.429850 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:47.429277 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:47.569294 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:47.569255 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:48.195982 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:48.195947 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:48.333258 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:48.333218 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:48.774030 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:48.773997 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:48.908341 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:48.908265 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:48.950550 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:48.947676 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb"] Oct 30 00:54:48.994755 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:48.992011 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.012907 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.012875 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-95492e-openshift-default-dockercfg-lz5kv\"" Oct 30 00:54:49.087819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.021177 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istiod-ca-cert\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.087819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.074908 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-credential-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.087819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.074954 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-podinfo\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.087819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.075006 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-certs\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.087819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.075078 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-envoy\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.087819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.075148 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-data\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.087819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.075178 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-token\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.087819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.075307 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2m2z\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-kube-api-access-f2m2z\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.087819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.075381 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.102803 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.102763 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb"] Oct 30 00:54:49.132450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.131425 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:49.344796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.319427 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.345191 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.345162 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istiod-ca-cert\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.345423 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.345404 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-credential-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.345565 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.345550 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-podinfo\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.402484 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.402448 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-certs\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.402845 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.402817 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-envoy\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.403117 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.403089 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-data\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.403289 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.403264 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-token\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.403586 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.403564 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2m2z\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-kube-api-access-f2m2z\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.404258 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.348212 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-credential-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.434737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.404698 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-data\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.436062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.428797 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-podinfo\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.436250 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.429060 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-certs\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.436356 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.346699 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istiod-ca-cert\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.436454 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.338167 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.477108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.477030 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-envoy\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.490440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.490370 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-token\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.499923 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.499853 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2m2z\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-kube-api-access-f2m2z\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.737451 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.733788 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:49.781154 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:49.781122 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:50.212650 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:50.212082 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb"] Oct 30 00:54:50.220686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:50.220638 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:54:50.220826 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:50.220765 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:54:50.220888 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:50.220871 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:54:50.498790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:50.497987 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" event={"ID":"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018","Type":"ContainerStarted","Data":"50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf"} Oct 30 00:54:50.498790 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:50.498058 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" event={"ID":"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018","Type":"ContainerStarted","Data":"c40ebf7449b8166c125902d42cf1561563104225ca9bc33d9be04b1cae951304"} Oct 30 00:54:50.547497 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:50.547433 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podStartSLOduration=2.547413721 podStartE2EDuration="2.547413721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:50.54246831 +0000 UTC m=+1228.454595448" watchObservedRunningTime="2026-10-30 00:54:50.547413721 +0000 UTC m=+1228.459540870" Oct 30 00:54:50.672643 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:50.671379 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:50.740728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:50.736813 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:51.740461 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:51.740423 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:51.845354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:51.845319 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:51.970500 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:51.970459 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:52.519646 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:52.519608 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:52.520605 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:52.520577 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:54:52.668408 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:52.668306 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:53.515845 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:53.515810 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:53.756970 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:53.756940 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:53.889871 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:53.889833 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:54.571470 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:54.569952 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": read tcp 10.133.0.2:44116->10.133.0.85:15021: read: connection reset by peer" start-of-body= Oct 30 00:54:54.571470 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:54.570034 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": read tcp 10.133.0.2:44116->10.133.0.85:15021: read: connection reset by peer" Oct 30 00:54:54.936596 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:54.936505 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:55.546956 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:55.546921 4517 generic.go:356] "Generic (PLEG): container finished" podID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerID="d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621" exitCode=0 Oct 30 00:54:55.547180 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:55.547001 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" event={"ID":"35e1387e-0825-4d7d-b5c5-e1282d033a11","Type":"ContainerDied","Data":"d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621"} Oct 30 00:54:55.547542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:55.547524 4517 scope.go:122] "RemoveContainer" containerID="d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621" Oct 30 00:54:55.549231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:55.549199 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:54:55.549348 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:55.549282 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:54:55.549348 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:55.549334 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:54:56.091738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:56.091605 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:56.213054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:56.213026 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:56.598734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:56.597887 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" event={"ID":"35e1387e-0825-4d7d-b5c5-e1282d033a11","Type":"ContainerStarted","Data":"b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273"} Oct 30 00:54:57.130024 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.129985 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:57.154135 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.154105 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8991-73f55f"] Oct 30 00:54:57.161735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.158370 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-73f55f" Oct 30 00:54:57.189154 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.187939 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v4hbx\"" Oct 30 00:54:57.231323 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.231293 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:54:57.235985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.235607 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:54:57.236250 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.236221 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 00:54:57.238497 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.238440 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-73f55f"] Oct 30 00:54:57.238640 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.238579 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmsd\" (UniqueName: \"kubernetes.io/projected/ba22691b-211f-448d-b374-03f60de9ae71-kube-api-access-kwmsd\") pod \"test-curl-8991-73f55f\" (UID: \"ba22691b-211f-448d-b374-03f60de9ae71\") " pod="opendatahub/test-curl-8991-73f55f" Oct 30 00:54:57.340743 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.340694 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmsd\" (UniqueName: \"kubernetes.io/projected/ba22691b-211f-448d-b374-03f60de9ae71-kube-api-access-kwmsd\") pod \"test-curl-8991-73f55f\" (UID: \"ba22691b-211f-448d-b374-03f60de9ae71\") " pod="opendatahub/test-curl-8991-73f55f" Oct 30 00:54:57.397047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.396550 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmsd\" (UniqueName: \"kubernetes.io/projected/ba22691b-211f-448d-b374-03f60de9ae71-kube-api-access-kwmsd\") pod \"test-curl-8991-73f55f\" (UID: \"ba22691b-211f-448d-b374-03f60de9ae71\") " pod="opendatahub/test-curl-8991-73f55f" Oct 30 00:54:57.536393 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.536361 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-73f55f" Oct 30 00:54:57.893606 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:57.893570 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-73f55f"] Oct 30 00:54:57.895941 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:54:57.895912 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba22691b_211f_448d_b374_03f60de9ae71.slice/crio-ba95e5dd514471095cde19cad3f43705bc9bb07bb65839a1838911f2112bad8e WatchSource:0}: Error finding container ba95e5dd514471095cde19cad3f43705bc9bb07bb65839a1838911f2112bad8e: Status 404 returned error can't find the container with id ba95e5dd514471095cde19cad3f43705bc9bb07bb65839a1838911f2112bad8e Oct 30 00:54:58.216820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:58.216734 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:54:58.216820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:58.216795 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 00:54:58.617822 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:58.617775 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-73f55f" event={"ID":"ba22691b-211f-448d-b374-03f60de9ae71","Type":"ContainerStarted","Data":"a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9"} Oct 30 00:54:58.617822 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:58.617830 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-73f55f" event={"ID":"ba22691b-211f-448d-b374-03f60de9ae71","Type":"ContainerStarted","Data":"ba95e5dd514471095cde19cad3f43705bc9bb07bb65839a1838911f2112bad8e"} Oct 30 00:54:58.653266 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:58.653217 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8991-73f55f" podStartSLOduration=1.65320315 podStartE2EDuration="1.65320315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:58.652160833 +0000 UTC m=+1236.564287974" watchObservedRunningTime="2026-10-30 00:54:58.65320315 +0000 UTC m=+1236.565330290" Oct 30 00:54:58.885786 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:58.885294 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:59.216829 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.216701 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:54:59.216829 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.216800 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 00:54:59.594681 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.594648 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8991-73f55f"] Oct 30 00:54:59.602251 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.601535 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8991-73f55f"] Oct 30 00:54:59.719596 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.719565 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8991-1ed87b"] Oct 30 00:54:59.720279 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.720256 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ba22691b-211f-448d-b374-03f60de9ae71" containerName="test-curl-8991-73f55f" Oct 30 00:54:59.720396 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.720280 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba22691b-211f-448d-b374-03f60de9ae71" containerName="test-curl-8991-73f55f" Oct 30 00:54:59.720454 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.720428 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba22691b-211f-448d-b374-03f60de9ae71" containerName="test-curl-8991-73f55f" Oct 30 00:54:59.724941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.724918 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-1ed87b" Oct 30 00:54:59.742900 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.742874 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-1ed87b"] Oct 30 00:54:59.800373 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.800342 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdq7t\" (UniqueName: \"kubernetes.io/projected/bd6d4bff-637a-4818-b5c5-765e5a4afc3a-kube-api-access-pdq7t\") pod \"test-curl-8991-1ed87b\" (UID: \"bd6d4bff-637a-4818-b5c5-765e5a4afc3a\") " pod="opendatahub/test-curl-8991-1ed87b" Oct 30 00:54:59.902157 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.902069 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdq7t\" (UniqueName: \"kubernetes.io/projected/bd6d4bff-637a-4818-b5c5-765e5a4afc3a-kube-api-access-pdq7t\") pod \"test-curl-8991-1ed87b\" (UID: \"bd6d4bff-637a-4818-b5c5-765e5a4afc3a\") " pod="opendatahub/test-curl-8991-1ed87b" Oct 30 00:54:59.923327 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:54:59.923298 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdq7t\" (UniqueName: \"kubernetes.io/projected/bd6d4bff-637a-4818-b5c5-765e5a4afc3a-kube-api-access-pdq7t\") pod \"test-curl-8991-1ed87b\" (UID: \"bd6d4bff-637a-4818-b5c5-765e5a4afc3a\") " pod="opendatahub/test-curl-8991-1ed87b" Oct 30 00:55:00.041901 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:00.041868 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-1ed87b" Oct 30 00:55:00.216982 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:00.216873 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:55:00.216982 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:00.216931 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 00:55:00.385982 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:00.385521 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-1ed87b"] Oct 30 00:55:00.388714 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:00.388678 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd6d4bff_637a_4818_b5c5_765e5a4afc3a.slice/crio-869b8c237abf2658b0560702cb6f35045ce185abf519f42fc45fb3e9fdf42511 WatchSource:0}: Error finding container 869b8c237abf2658b0560702cb6f35045ce185abf519f42fc45fb3e9fdf42511: Status 404 returned error can't find the container with id 869b8c237abf2658b0560702cb6f35045ce185abf519f42fc45fb3e9fdf42511 Oct 30 00:55:00.635795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:00.635752 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-1ed87b" event={"ID":"bd6d4bff-637a-4818-b5c5-765e5a4afc3a","Type":"ContainerStarted","Data":"ffed6339348e867cac4ce2d02df16d508aa2646045cdda3f78cd683af9bda9bc"} Oct 30 00:55:00.635795 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:00.635801 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-1ed87b" event={"ID":"bd6d4bff-637a-4818-b5c5-765e5a4afc3a","Type":"ContainerStarted","Data":"869b8c237abf2658b0560702cb6f35045ce185abf519f42fc45fb3e9fdf42511"} Oct 30 00:55:00.636083 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:00.635931 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8991-73f55f" podUID="ba22691b-211f-448d-b374-03f60de9ae71" containerName="test-curl-8991-73f55f" containerID="cri-o://a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9" gracePeriod=2 Oct 30 00:55:00.687736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:00.687333 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8991-1ed87b" podStartSLOduration=1.687312629 podStartE2EDuration="1.687312629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:00.684929024 +0000 UTC m=+1238.597056166" watchObservedRunningTime="2026-10-30 00:55:00.687312629 +0000 UTC m=+1238.599439772" Oct 30 00:55:00.718548 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:00.718518 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:00.803734 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:55:00.802976 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:55:01.217275 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.217188 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 00:55:01.217275 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.217258 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 00:55:01.619741 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.619659 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8991-1ed87b"] Oct 30 00:55:01.624528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.624496 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8991-1ed87b"] Oct 30 00:55:01.788438 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.788403 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8991-3c1955"] Oct 30 00:55:01.788988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.788974 4517 cpu_manager.go:414] "containerMap: removing container" podUID="bd6d4bff-637a-4818-b5c5-765e5a4afc3a" containerName="test-curl-8991-1ed87b" Oct 30 00:55:01.789048 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.788989 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6d4bff-637a-4818-b5c5-765e5a4afc3a" containerName="test-curl-8991-1ed87b" Oct 30 00:55:01.789086 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.789082 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6d4bff-637a-4818-b5c5-765e5a4afc3a" containerName="test-curl-8991-1ed87b" Oct 30 00:55:01.792760 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.792739 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-3c1955" Oct 30 00:55:01.819043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.815569 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-3c1955"] Oct 30 00:55:01.930641 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:01.930554 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2pt5\" (UniqueName: \"kubernetes.io/projected/37a76594-f8f8-456b-b749-b9fb02bacc9d-kube-api-access-n2pt5\") pod \"test-curl-8991-3c1955\" (UID: \"37a76594-f8f8-456b-b749-b9fb02bacc9d\") " pod="opendatahub/test-curl-8991-3c1955" Oct 30 00:55:02.032939 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.032889 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2pt5\" (UniqueName: \"kubernetes.io/projected/37a76594-f8f8-456b-b749-b9fb02bacc9d-kube-api-access-n2pt5\") pod \"test-curl-8991-3c1955\" (UID: \"37a76594-f8f8-456b-b749-b9fb02bacc9d\") " pod="opendatahub/test-curl-8991-3c1955" Oct 30 00:55:02.054554 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.054524 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2pt5\" (UniqueName: \"kubernetes.io/projected/37a76594-f8f8-456b-b749-b9fb02bacc9d-kube-api-access-n2pt5\") pod \"test-curl-8991-3c1955\" (UID: \"37a76594-f8f8-456b-b749-b9fb02bacc9d\") " pod="opendatahub/test-curl-8991-3c1955" Oct 30 00:55:02.116354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.116317 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-3c1955" Oct 30 00:55:02.254108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.253601 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:55:02.254108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.254082 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:55:02.255226 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.255110 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 00:55:02.441898 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.429115 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:02.459531 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.459438 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-3c1955"] Oct 30 00:55:02.652838 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.652805 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-3c1955" event={"ID":"37a76594-f8f8-456b-b749-b9fb02bacc9d","Type":"ContainerStarted","Data":"2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97"} Oct 30 00:55:02.652967 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.652849 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-3c1955" event={"ID":"37a76594-f8f8-456b-b749-b9fb02bacc9d","Type":"ContainerStarted","Data":"8ff69ef397a18a7766659a58cce16ae34f34a77b08d7ccd72dff7d48d6264ef4"} Oct 30 00:55:02.652967 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.652924 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8991-1ed87b" podUID="bd6d4bff-637a-4818-b5c5-765e5a4afc3a" containerName="test-curl-8991-1ed87b" containerID="cri-o://ffed6339348e867cac4ce2d02df16d508aa2646045cdda3f78cd683af9bda9bc" gracePeriod=2 Oct 30 00:55:02.691820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.691764 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8991-3c1955" podStartSLOduration=1.691749915 podStartE2EDuration="1.691749915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:02.687590403 +0000 UTC m=+1240.599717543" watchObservedRunningTime="2026-10-30 00:55:02.691749915 +0000 UTC m=+1240.603877057" Oct 30 00:55:02.806148 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.806122 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-73f55f" Oct 30 00:55:02.968799 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.968759 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba22691b-211f-448d-b374-03f60de9ae71-kube-api-access-kwmsd\" (UniqueName: \"kubernetes.io/projected/ba22691b-211f-448d-b374-03f60de9ae71-kube-api-access-kwmsd\") pod \"ba22691b-211f-448d-b374-03f60de9ae71\" (UID: \"ba22691b-211f-448d-b374-03f60de9ae71\") " Oct 30 00:55:02.971137 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:02.971103 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba22691b-211f-448d-b374-03f60de9ae71-kube-api-access-kwmsd" pod "ba22691b-211f-448d-b374-03f60de9ae71" (UID: "ba22691b-211f-448d-b374-03f60de9ae71"). InnerVolumeSpecName "kube-api-access-kwmsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:03.070734 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.070689 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwmsd\" (UniqueName: \"kubernetes.io/projected/ba22691b-211f-448d-b374-03f60de9ae71-kube-api-access-kwmsd\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:55:03.564926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.564880 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8991-3c1955"] Oct 30 00:55:03.570341 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.570314 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8991-3c1955"] Oct 30 00:55:03.664579 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.664537 4517 generic.go:356] "Generic (PLEG): container finished" podID="ba22691b-211f-448d-b374-03f60de9ae71" containerID="a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9" exitCode=137 Oct 30 00:55:03.664774 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.664585 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-73f55f" Oct 30 00:55:03.664774 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.664640 4517 scope.go:122] "RemoveContainer" containerID="a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9" Oct 30 00:55:03.674625 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.674603 4517 scope.go:122] "RemoveContainer" containerID="a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9" Oct 30 00:55:03.674892 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:55:03.674871 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9\": container with ID starting with a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9 not found: ID does not exist" containerID="a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9" Oct 30 00:55:03.674965 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.674900 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9"} err="failed to get container status \"a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9\": rpc error: code = NotFound desc = could not find container \"a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9\": container with ID starting with a38a78097bf01c45577d7e5eae6d0e3d860c822a3a7b8a5db59b8efdd1f28ec9 not found: ID does not exist" Oct 30 00:55:03.685377 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:55:03.685346 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8991-73f55f\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="ba22691b-211f-448d-b374-03f60de9ae71" pod="opendatahub/test-curl-8991-73f55f" Oct 30 00:55:03.725072 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.725035 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8991-dc316e"] Oct 30 00:55:03.725693 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.725671 4517 cpu_manager.go:414] "containerMap: removing container" podUID="37a76594-f8f8-456b-b749-b9fb02bacc9d" containerName="test-curl-8991-3c1955" Oct 30 00:55:03.725815 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.725693 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a76594-f8f8-456b-b749-b9fb02bacc9d" containerName="test-curl-8991-3c1955" Oct 30 00:55:03.725884 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.725860 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a76594-f8f8-456b-b749-b9fb02bacc9d" containerName="test-curl-8991-3c1955" Oct 30 00:55:03.732347 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.732326 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-dc316e" Oct 30 00:55:03.744873 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.744816 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-dc316e"] Oct 30 00:55:03.745008 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:55:03.744911 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8991-73f55f\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="ba22691b-211f-448d-b374-03f60de9ae71" pod="opendatahub/test-curl-8991-73f55f" Oct 30 00:55:03.884056 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.883968 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftx9d\" (UniqueName: \"kubernetes.io/projected/a5c2301a-268a-4881-8003-071df59f0ee8-kube-api-access-ftx9d\") pod \"test-curl-8991-dc316e\" (UID: \"a5c2301a-268a-4881-8003-071df59f0ee8\") " pod="opendatahub/test-curl-8991-dc316e" Oct 30 00:55:03.985752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:03.985700 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftx9d\" (UniqueName: \"kubernetes.io/projected/a5c2301a-268a-4881-8003-071df59f0ee8-kube-api-access-ftx9d\") pod \"test-curl-8991-dc316e\" (UID: \"a5c2301a-268a-4881-8003-071df59f0ee8\") " pod="opendatahub/test-curl-8991-dc316e" Oct 30 00:55:04.007334 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.007303 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftx9d\" (UniqueName: \"kubernetes.io/projected/a5c2301a-268a-4881-8003-071df59f0ee8-kube-api-access-ftx9d\") pod \"test-curl-8991-dc316e\" (UID: \"a5c2301a-268a-4881-8003-071df59f0ee8\") " pod="opendatahub/test-curl-8991-dc316e" Oct 30 00:55:04.028932 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.028903 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:04.049564 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.049538 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-dc316e" Oct 30 00:55:04.237059 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.237010 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba22691b-211f-448d-b374-03f60de9ae71" path="/var/lib/kubelet/pods/ba22691b-211f-448d-b374-03f60de9ae71/volumes" Oct 30 00:55:04.363723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.363682 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-dc316e"] Oct 30 00:55:04.366069 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:04.366041 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5c2301a_268a_4881_8003_071df59f0ee8.slice/crio-afd3a0eabe9e6f0f73b4e02d053bc23e045aff26f8b85acb10bc7475860fcdf3 WatchSource:0}: Error finding container afd3a0eabe9e6f0f73b4e02d053bc23e045aff26f8b85acb10bc7475860fcdf3: Status 404 returned error can't find the container with id afd3a0eabe9e6f0f73b4e02d053bc23e045aff26f8b85acb10bc7475860fcdf3 Oct 30 00:55:04.676239 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.676210 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-dc316e" event={"ID":"a5c2301a-268a-4881-8003-071df59f0ee8","Type":"ContainerStarted","Data":"b277bb988b00899607de5b816d7d13b8edeb9ebfc5712f110b19d1669362ff09"} Oct 30 00:55:04.676617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.676255 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-dc316e" event={"ID":"a5c2301a-268a-4881-8003-071df59f0ee8","Type":"ContainerStarted","Data":"afd3a0eabe9e6f0f73b4e02d053bc23e045aff26f8b85acb10bc7475860fcdf3"} Oct 30 00:55:04.678051 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.678023 4517 generic.go:356] "Generic (PLEG): container finished" podID="bd6d4bff-637a-4818-b5c5-765e5a4afc3a" containerID="ffed6339348e867cac4ce2d02df16d508aa2646045cdda3f78cd683af9bda9bc" exitCode=137 Oct 30 00:55:04.679473 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.679450 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8991-3c1955" podUID="37a76594-f8f8-456b-b749-b9fb02bacc9d" containerName="test-curl-8991-3c1955" containerID="cri-o://2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97" gracePeriod=2 Oct 30 00:55:04.713988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.713938 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8991-dc316e" podStartSLOduration=1.7139239800000001 podStartE2EDuration="1.71392398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:04.711281109 +0000 UTC m=+1242.623408268" watchObservedRunningTime="2026-10-30 00:55:04.71392398 +0000 UTC m=+1242.626051115" Oct 30 00:55:04.852702 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:04.852678 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-1ed87b" Oct 30 00:55:05.003835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:05.003736 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6d4bff-637a-4818-b5c5-765e5a4afc3a-kube-api-access-pdq7t\" (UniqueName: \"kubernetes.io/projected/bd6d4bff-637a-4818-b5c5-765e5a4afc3a-kube-api-access-pdq7t\") pod \"bd6d4bff-637a-4818-b5c5-765e5a4afc3a\" (UID: \"bd6d4bff-637a-4818-b5c5-765e5a4afc3a\") " Oct 30 00:55:05.006022 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:05.005996 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6d4bff-637a-4818-b5c5-765e5a4afc3a-kube-api-access-pdq7t" pod "bd6d4bff-637a-4818-b5c5-765e5a4afc3a" (UID: "bd6d4bff-637a-4818-b5c5-765e5a4afc3a"). InnerVolumeSpecName "kube-api-access-pdq7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:05.105700 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:05.105669 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdq7t\" (UniqueName: \"kubernetes.io/projected/bd6d4bff-637a-4818-b5c5-765e5a4afc3a-kube-api-access-pdq7t\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:55:05.609334 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:05.609300 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8991-dc316e"] Oct 30 00:55:05.614200 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:05.614167 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8991-dc316e"] Oct 30 00:55:05.689378 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:05.689345 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-1ed87b" Oct 30 00:55:05.689808 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:05.689390 4517 scope.go:122] "RemoveContainer" containerID="ffed6339348e867cac4ce2d02df16d508aa2646045cdda3f78cd683af9bda9bc" Oct 30 00:55:05.714383 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:05.714357 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:05.717559 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:55:05.717536 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8991-1ed87b\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="bd6d4bff-637a-4818-b5c5-765e5a4afc3a" pod="opendatahub/test-curl-8991-1ed87b" Oct 30 00:55:06.154332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.154301 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:06.236398 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.236360 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6d4bff-637a-4818-b5c5-765e5a4afc3a" path="/var/lib/kubelet/pods/bd6d4bff-637a-4818-b5c5-765e5a4afc3a/volumes" Oct 30 00:55:06.574385 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.574341 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd"] Oct 30 00:55:06.579974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.575374 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a5c2301a-268a-4881-8003-071df59f0ee8" containerName="test-curl-8991-dc316e" Oct 30 00:55:06.579974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.575400 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c2301a-268a-4881-8003-071df59f0ee8" containerName="test-curl-8991-dc316e" Oct 30 00:55:06.579974 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.575552 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c2301a-268a-4881-8003-071df59f0ee8" containerName="test-curl-8991-dc316e" Oct 30 00:55:06.588775 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.582112 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.609014 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.608939 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd"] Oct 30 00:55:06.620452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.620423 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6867b8-openshift-default-dockercfg-r2nhs\"" Oct 30 00:55:06.623644 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.623619 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:06.701652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.701522 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8991-dc316e" podUID="a5c2301a-268a-4881-8003-071df59f0ee8" containerName="test-curl-8991-dc316e" containerID="cri-o://b277bb988b00899607de5b816d7d13b8edeb9ebfc5712f110b19d1669362ff09" gracePeriod=2 Oct 30 00:55:06.730363 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.730330 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-podinfo\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.730553 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.730376 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-token\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.730553 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.730401 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istiod-ca-cert\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.730686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.730601 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-data\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.730769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.730680 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-certs\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.730822 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.730789 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vmf\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-kube-api-access-n6vmf\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.730873 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.730823 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-envoy\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.730926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.730901 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-socket\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.731062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.731026 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-credential-socket\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.767736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.766381 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:06.842778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.841655 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-socket\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.842778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.842141 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-socket\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.842778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.842316 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-credential-socket\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.842778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.842607 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-credential-socket\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.843129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.843000 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-podinfo\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.849744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.843546 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-token\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.849744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.843900 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istiod-ca-cert\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.849744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.844594 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istiod-ca-cert\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.849744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.846349 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-data\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.849744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.846662 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-data\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.849744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.847193 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-certs\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.849744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.847239 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-certs\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.849744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.847332 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vmf\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-kube-api-access-n6vmf\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.849744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.847673 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-envoy\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.860747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.860397 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-envoy\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.885341 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.878550 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-podinfo\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.894032 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.893998 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-token\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.909752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.905688 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vmf\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-kube-api-access-n6vmf\") pod \"e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:06.923801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:06.923496 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:07.056641 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.046924 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-3c1955" Oct 30 00:55:07.100545 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.100452 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a76594-f8f8-456b-b749-b9fb02bacc9d-kube-api-access-n2pt5\" (UniqueName: \"kubernetes.io/projected/37a76594-f8f8-456b-b749-b9fb02bacc9d-kube-api-access-n2pt5\") pod \"37a76594-f8f8-456b-b749-b9fb02bacc9d\" (UID: \"37a76594-f8f8-456b-b749-b9fb02bacc9d\") " Oct 30 00:55:07.126020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.125769 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a76594-f8f8-456b-b749-b9fb02bacc9d-kube-api-access-n2pt5" pod "37a76594-f8f8-456b-b749-b9fb02bacc9d" (UID: "37a76594-f8f8-456b-b749-b9fb02bacc9d"). InnerVolumeSpecName "kube-api-access-n2pt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:07.202375 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.202325 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2pt5\" (UniqueName: \"kubernetes.io/projected/37a76594-f8f8-456b-b749-b9fb02bacc9d-kube-api-access-n2pt5\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:55:07.446547 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.446516 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd"] Oct 30 00:55:07.447563 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:07.447533 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcae3bbb_d666_405f_8a3b_e7b2440711a8.slice/crio-64983cc980e9f00523a4978803782ff52c7619bdbdb78b545b5e34e251cedcf5 WatchSource:0}: Error finding container 64983cc980e9f00523a4978803782ff52c7619bdbdb78b545b5e34e251cedcf5: Status 404 returned error can't find the container with id 64983cc980e9f00523a4978803782ff52c7619bdbdb78b545b5e34e251cedcf5 Oct 30 00:55:07.450940 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.450897 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:55:07.451046 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.450988 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:55:07.451091 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.451032 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:55:07.635443 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.635407 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:07.770792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.770687 4517 generic.go:356] "Generic (PLEG): container finished" podID="37a76594-f8f8-456b-b749-b9fb02bacc9d" containerID="2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97" exitCode=137 Oct 30 00:55:07.771317 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.770871 4517 scope.go:122] "RemoveContainer" containerID="2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97" Oct 30 00:55:07.771317 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.770947 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-3c1955" Oct 30 00:55:07.780054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.780012 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" event={"ID":"dcae3bbb-d666-405f-8a3b-e7b2440711a8","Type":"ContainerStarted","Data":"ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3"} Oct 30 00:55:07.780233 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.780121 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" event={"ID":"dcae3bbb-d666-405f-8a3b-e7b2440711a8","Type":"ContainerStarted","Data":"64983cc980e9f00523a4978803782ff52c7619bdbdb78b545b5e34e251cedcf5"} Oct 30 00:55:07.796075 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.796050 4517 scope.go:122] "RemoveContainer" containerID="2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97" Oct 30 00:55:07.796906 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:55:07.796872 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97\": container with ID starting with 2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97 not found: ID does not exist" containerID="2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97" Oct 30 00:55:07.797043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.796919 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97"} err="failed to get container status \"2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97\": rpc error: code = NotFound desc = could not find container \"2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97\": container with ID starting with 2cdd9c5c9136549861db4666923e7b1c53dcf58e9600a0384b6b02229cb1ec97 not found: ID does not exist" Oct 30 00:55:07.843456 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.843401 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" podStartSLOduration=1.84338772 podStartE2EDuration="1.84338772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:07.832879487 +0000 UTC m=+1245.745006640" watchObservedRunningTime="2026-10-30 00:55:07.84338772 +0000 UTC m=+1245.755514860" Oct 30 00:55:07.844772 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:55:07.844679 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8991-3c1955\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="37a76594-f8f8-456b-b749-b9fb02bacc9d" pod="opendatahub/test-curl-8991-3c1955" Oct 30 00:55:07.925085 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:07.925044 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:08.236437 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:08.236402 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a76594-f8f8-456b-b749-b9fb02bacc9d" path="/var/lib/kubelet/pods/37a76594-f8f8-456b-b749-b9fb02bacc9d/volumes" Oct 30 00:55:08.789971 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:08.789922 4517 generic.go:356] "Generic (PLEG): container finished" podID="a5c2301a-268a-4881-8003-071df59f0ee8" containerID="b277bb988b00899607de5b816d7d13b8edeb9ebfc5712f110b19d1669362ff09" exitCode=137 Oct 30 00:55:08.931242 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:08.931213 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:08.938798 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:08.938770 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-dc316e" Oct 30 00:55:08.947116 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:08.947086 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c2301a-268a-4881-8003-071df59f0ee8-kube-api-access-ftx9d\" (UniqueName: \"kubernetes.io/projected/a5c2301a-268a-4881-8003-071df59f0ee8-kube-api-access-ftx9d\") pod \"a5c2301a-268a-4881-8003-071df59f0ee8\" (UID: \"a5c2301a-268a-4881-8003-071df59f0ee8\") " Oct 30 00:55:08.949894 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:08.949856 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c2301a-268a-4881-8003-071df59f0ee8-kube-api-access-ftx9d" pod "a5c2301a-268a-4881-8003-071df59f0ee8" (UID: "a5c2301a-268a-4881-8003-071df59f0ee8"). InnerVolumeSpecName "kube-api-access-ftx9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:09.048433 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:09.048392 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftx9d\" (UniqueName: \"kubernetes.io/projected/a5c2301a-268a-4881-8003-071df59f0ee8-kube-api-access-ftx9d\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:55:09.798474 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:09.798443 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-dc316e" Oct 30 00:55:09.798988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:09.798502 4517 scope.go:122] "RemoveContainer" containerID="b277bb988b00899607de5b816d7d13b8edeb9ebfc5712f110b19d1669362ff09" Oct 30 00:55:09.798988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:09.798836 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:09.800201 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:09.800176 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:55:09.815087 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:09.815057 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:09.864302 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:55:09.864053 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8991-dc316e\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="a5c2301a-268a-4881-8003-071df59f0ee8" pod="opendatahub/test-curl-8991-dc316e" Oct 30 00:55:10.236757 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:10.236662 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c2301a-268a-4881-8003-071df59f0ee8" path="/var/lib/kubelet/pods/a5c2301a-268a-4881-8003-071df59f0ee8/volumes" Oct 30 00:55:10.851538 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:10.851502 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:10.985992 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:10.985961 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:11.707875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:11.707846 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:11.823642 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:11.820831 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:12.127787 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.127723 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:12.571054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.570998 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:12.697680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.697628 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:12.734071 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.734045 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7"] Oct 30 00:55:12.740657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.740629 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:12.760639 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.760600 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7"] Oct 30 00:55:12.765332 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.765305 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6867b8-dockercfg-zlkpw\"" Oct 30 00:55:12.786039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.786003 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6867b8\"" Oct 30 00:55:12.897823 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.891168 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tg2q\" (UniqueName: \"kubernetes.io/projected/3b9611dd-f57d-407d-a795-ca22244f4ef6-kube-api-access-7tg2q\") pod \"payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7\" (UID: \"3b9611dd-f57d-407d-a795-ca22244f4ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:12.897823 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.891487 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9611dd-f57d-407d-a795-ca22244f4ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7\" (UID: \"3b9611dd-f57d-407d-a795-ca22244f4ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:12.997989 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.997948 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9611dd-f57d-407d-a795-ca22244f4ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7\" (UID: \"3b9611dd-f57d-407d-a795-ca22244f4ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:12.998307 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:12.998287 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tg2q\" (UniqueName: \"kubernetes.io/projected/3b9611dd-f57d-407d-a795-ca22244f4ef6-kube-api-access-7tg2q\") pod \"payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7\" (UID: \"3b9611dd-f57d-407d-a795-ca22244f4ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:13.034726 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:13.007818 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9611dd-f57d-407d-a795-ca22244f4ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7\" (UID: \"3b9611dd-f57d-407d-a795-ca22244f4ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:13.078577 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:13.076581 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tg2q\" (UniqueName: \"kubernetes.io/projected/3b9611dd-f57d-407d-a795-ca22244f4ef6-kube-api-access-7tg2q\") pod \"payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7\" (UID: \"3b9611dd-f57d-407d-a795-ca22244f4ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:13.362214 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:13.362177 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:13.714768 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:13.711105 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9611dd_f57d_407d_a795_ca22244f4ef6.slice/crio-125737ea0bbbe750a04645e4c0ebc03b624eaeb811ef79dc88d7fc138780080b WatchSource:0}: Error finding container 125737ea0bbbe750a04645e4c0ebc03b624eaeb811ef79dc88d7fc138780080b: Status 404 returned error can't find the container with id 125737ea0bbbe750a04645e4c0ebc03b624eaeb811ef79dc88d7fc138780080b Oct 30 00:55:13.714768 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:13.711400 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7"] Oct 30 00:55:14.067368 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:14.067329 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" event={"ID":"3b9611dd-f57d-407d-a795-ca22244f4ef6","Type":"ContainerStarted","Data":"8bb71df0ac6fbd7e24e613d8d0e25925d05d8c2996d914e5f08b3775cb156737"} Oct 30 00:55:14.067604 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:14.067394 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" event={"ID":"3b9611dd-f57d-407d-a795-ca22244f4ef6","Type":"ContainerStarted","Data":"125737ea0bbbe750a04645e4c0ebc03b624eaeb811ef79dc88d7fc138780080b"} Oct 30 00:55:14.067604 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:14.067554 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:14.071463 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:14.071434 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:14.109220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:14.109154 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" podStartSLOduration=2.10913232 podStartE2EDuration="2.10913232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:14.103674562 +0000 UTC m=+1252.015801704" watchObservedRunningTime="2026-10-30 00:55:14.10913232 +0000 UTC m=+1252.021259463" Oct 30 00:55:14.655319 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:14.654617 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:14.970500 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:14.970324 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:14.992439 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:14.992401 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt"] Oct 30 00:55:14.997187 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:14.997161 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.020851 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.020819 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d8090513\"/\"default-dockercfg-h5nxj\"" Oct 30 00:55:15.021062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.020926 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d8090513\"/\"openshift-service-ca.crt\"" Oct 30 00:55:15.021062 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.021039 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d8090513\"/\"e2e-lifecycle-model-d8090513-kserve-self-signed-certs\"" Oct 30 00:55:15.021204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.021131 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d8090513\"/\"kube-root-ca.crt\"" Oct 30 00:55:15.027215 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.027182 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt"] Oct 30 00:55:15.037698 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.037648 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-home\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.038084 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.038059 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b8e0b5-d0bf-4a50-b448-de3117133944-tls-certs\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.049911 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.049875 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-dshm\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.050345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.050317 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gkfr\" (UniqueName: \"kubernetes.io/projected/36b8e0b5-d0bf-4a50-b448-de3117133944-kube-api-access-2gkfr\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.050534 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.050503 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-model-cache\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.050873 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.050848 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-tmp-dir\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.153804 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.153750 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gkfr\" (UniqueName: \"kubernetes.io/projected/36b8e0b5-d0bf-4a50-b448-de3117133944-kube-api-access-2gkfr\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.173958 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.173915 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-model-cache\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.174350 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.174327 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-tmp-dir\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.174517 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.174485 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-home\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.174784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.174764 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b8e0b5-d0bf-4a50-b448-de3117133944-tls-certs\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.174931 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.174898 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-model-cache\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.175178 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.174954 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-dshm\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.175582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.175551 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-tmp-dir\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.176521 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.176488 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-home\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.230528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.230441 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gkfr\" (UniqueName: \"kubernetes.io/projected/36b8e0b5-d0bf-4a50-b448-de3117133944-kube-api-access-2gkfr\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.230756 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.230652 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-dshm\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.231259 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.231236 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b8e0b5-d0bf-4a50-b448-de3117133944-tls-certs\") pod \"e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.313263 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.313170 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:15.684124 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.684091 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt"] Oct 30 00:55:15.685891 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:15.685866 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36b8e0b5_d0bf_4a50_b448_de3117133944.slice/crio-1bde5297f9e0ebd82e3e95b0713f8280f87058fe773ab178ef2d06e668520c14 WatchSource:0}: Error finding container 1bde5297f9e0ebd82e3e95b0713f8280f87058fe773ab178ef2d06e668520c14: Status 404 returned error can't find the container with id 1bde5297f9e0ebd82e3e95b0713f8280f87058fe773ab178ef2d06e668520c14 Oct 30 00:55:15.769027 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.768331 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:15.884731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:15.884053 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:16.040050 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:16.040020 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:16.102923 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:16.102883 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" event={"ID":"36b8e0b5-d0bf-4a50-b448-de3117133944","Type":"ContainerStarted","Data":"1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113"} Oct 30 00:55:16.103133 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:16.102937 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" event={"ID":"36b8e0b5-d0bf-4a50-b448-de3117133944","Type":"ContainerStarted","Data":"1bde5297f9e0ebd82e3e95b0713f8280f87058fe773ab178ef2d06e668520c14"} Oct 30 00:55:16.142693 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:16.142635 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" podStartSLOduration=2.142621558 podStartE2EDuration="2.142621558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:16.139876353 +0000 UTC m=+1254.052003493" watchObservedRunningTime="2026-10-30 00:55:16.142621558 +0000 UTC m=+1254.054748735" Oct 30 00:55:16.315530 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:16.314411 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:17.533467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:17.533433 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:19.226655 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:19.226621 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:19.897141 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:19.897099 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:21.029635 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:21.029598 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:23.271447 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:23.271419 4517 scope.go:122] "RemoveContainer" containerID="2b72bc2e6aaa2ffc5b1fb3f107fde7c30b1798148bd43b732c6098fd72eb94e0" Oct 30 00:55:23.446589 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:23.446523 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:25.095555 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:25.095523 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:55:25.287155 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:25.287125 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:25.313908 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:25.313648 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:25.344692 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:25.344662 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:26.306216 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:26.306187 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 00:55:27.027879 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:27.027839 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:29.008838 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:29.008805 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:29.134036 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:29.133995 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:29.555730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:29.554980 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:30.028729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:30.028611 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:30.771874 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:55:30.771826 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:55:30.923573 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:30.923531 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:32.727465 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:32.727431 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:33.304389 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:33.304355 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:33.445057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:33.445023 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:34.582867 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.582837 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n"] Oct 30 00:55:34.588541 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.588519 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.605962 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.605931 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6867b8-serving-cert\"" Oct 30 00:55:34.624633 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.624588 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n"] Oct 30 00:55:34.688244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.688207 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:34.728869 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.728831 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-metrics-certs\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.729074 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.729000 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71412d84-5e97-4794-aea9-4418023285d0-openshift-service-ca\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.729074 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.729071 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmwh\" (UniqueName: \"kubernetes.io/projected/71412d84-5e97-4794-aea9-4418023285d0-kube-api-access-rsmwh\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.729231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.729207 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-maas-api-tls\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.805821 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.805783 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:34.831231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.831189 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmwh\" (UniqueName: \"kubernetes.io/projected/71412d84-5e97-4794-aea9-4418023285d0-kube-api-access-rsmwh\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.831425 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.831305 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-maas-api-tls\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.831425 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.831400 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-metrics-certs\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.831599 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.831576 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71412d84-5e97-4794-aea9-4418023285d0-openshift-service-ca\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.832271 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.832240 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71412d84-5e97-4794-aea9-4418023285d0-openshift-service-ca\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.834234 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.834177 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-metrics-certs\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.834356 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.834332 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-maas-api-tls\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.853433 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.853398 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmwh\" (UniqueName: \"kubernetes.io/projected/71412d84-5e97-4794-aea9-4418023285d0-kube-api-access-rsmwh\") pod \"maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:34.905313 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:34.905277 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:35.301503 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:35.301471 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n"] Oct 30 00:55:35.303323 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:35.303293 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71412d84_5e97_4794_aea9_4418023285d0.slice/crio-503f78ca3c9a33fbb8e1f11b4bc716e53a1dd3e93a682f3aa3aeb614f2e52c92 WatchSource:0}: Error finding container 503f78ca3c9a33fbb8e1f11b4bc716e53a1dd3e93a682f3aa3aeb614f2e52c92: Status 404 returned error can't find the container with id 503f78ca3c9a33fbb8e1f11b4bc716e53a1dd3e93a682f3aa3aeb614f2e52c92 Oct 30 00:55:35.360136 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:35.360100 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" event={"ID":"71412d84-5e97-4794-aea9-4418023285d0","Type":"ContainerStarted","Data":"503f78ca3c9a33fbb8e1f11b4bc716e53a1dd3e93a682f3aa3aeb614f2e52c92"} Oct 30 00:55:35.603617 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:35.603505 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:35.724373 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:35.724310 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:36.376424 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:36.375753 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" event={"ID":"71412d84-5e97-4794-aea9-4418023285d0","Type":"ContainerStarted","Data":"fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6"} Oct 30 00:55:36.378443 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:36.376911 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:36.420299 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:36.420249 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" podStartSLOduration=2.09550567 podStartE2EDuration="2.420233305s" totalImagesPullingTime="324.727635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:36.414679516 +0000 UTC m=+1274.326806659" watchObservedRunningTime="2026-10-30 00:55:36.420233305 +0000 UTC m=+1274.332360445" Oct 30 00:55:37.144519 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:37.144484 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:38.830230 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:38.830187 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:40.996886 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:40.996845 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:42.807384 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:42.807343 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:42.921941 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:42.921879 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:43.302015 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:43.301980 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:43.398330 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:43.398301 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:55:43.800777 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:43.798885 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:44.005733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:44.000098 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:45.104192 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:45.104045 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:47.015726 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:47.015669 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:47.559763 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:47.559730 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:47.684446 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:47.684409 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:48.046476 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:48.046433 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:48.551182 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:48.551138 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:48.667814 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:48.667771 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:48.964820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:48.964664 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:49.169425 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.169365 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:49.317933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.317901 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:49.444158 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.444089 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:49.494335 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.494294 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj"] Oct 30 00:55:49.500383 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.500359 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:55:49.515666 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.514528 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj"] Oct 30 00:55:49.523758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.522125 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-62f5d1-dockercfg-8657x\"" Oct 30 00:55:49.523758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.522273 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-62f5d1\"" Oct 30 00:55:49.619842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.618750 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:49.675730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.654941 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25333518-39ac-4e6c-a406-80a09d216b14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj\" (UID: \"25333518-39ac-4e6c-a406-80a09d216b14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:55:49.676495 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.676004 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ksf\" (UniqueName: \"kubernetes.io/projected/25333518-39ac-4e6c-a406-80a09d216b14-kube-api-access-v8ksf\") pod \"payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj\" (UID: \"25333518-39ac-4e6c-a406-80a09d216b14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:55:49.788222 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.788183 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ksf\" (UniqueName: \"kubernetes.io/projected/25333518-39ac-4e6c-a406-80a09d216b14-kube-api-access-v8ksf\") pod \"payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj\" (UID: \"25333518-39ac-4e6c-a406-80a09d216b14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:55:49.802924 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.802889 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25333518-39ac-4e6c-a406-80a09d216b14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj\" (UID: \"25333518-39ac-4e6c-a406-80a09d216b14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:55:49.804229 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.803993 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25333518-39ac-4e6c-a406-80a09d216b14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj\" (UID: \"25333518-39ac-4e6c-a406-80a09d216b14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:55:49.809593 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.809569 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx"] Oct 30 00:55:49.892108 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.892015 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:55:49.899772 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.899548 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx"] Oct 30 00:55:49.917426 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:49.916747 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ksf\" (UniqueName: \"kubernetes.io/projected/25333518-39ac-4e6c-a406-80a09d216b14-kube-api-access-v8ksf\") pod \"payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj\" (UID: \"25333518-39ac-4e6c-a406-80a09d216b14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:55:50.028582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.027843 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1885f34e-6420-4b73-aab4-18d173a0f95a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx\" (UID: \"1885f34e-6420-4b73-aab4-18d173a0f95a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:55:50.028582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.027912 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqqg\" (UniqueName: \"kubernetes.io/projected/1885f34e-6420-4b73-aab4-18d173a0f95a-kube-api-access-fdqqg\") pod \"payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx\" (UID: \"1885f34e-6420-4b73-aab4-18d173a0f95a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:55:50.154767 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.153397 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:55:50.154950 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.154804 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1885f34e-6420-4b73-aab4-18d173a0f95a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx\" (UID: \"1885f34e-6420-4b73-aab4-18d173a0f95a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:55:50.154950 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.154851 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqqg\" (UniqueName: \"kubernetes.io/projected/1885f34e-6420-4b73-aab4-18d173a0f95a-kube-api-access-fdqqg\") pod \"payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx\" (UID: \"1885f34e-6420-4b73-aab4-18d173a0f95a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:55:50.155918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.155867 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1885f34e-6420-4b73-aab4-18d173a0f95a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx\" (UID: \"1885f34e-6420-4b73-aab4-18d173a0f95a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:55:50.199212 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.199184 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqqg\" (UniqueName: \"kubernetes.io/projected/1885f34e-6420-4b73-aab4-18d173a0f95a-kube-api-access-fdqqg\") pod \"payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx\" (UID: \"1885f34e-6420-4b73-aab4-18d173a0f95a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:55:50.226784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.226029 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:55:50.741744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.740483 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj"] Oct 30 00:55:50.747204 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:50.747166 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25333518_39ac_4e6c_a406_80a09d216b14.slice/crio-12ee10c97e4489286741ede947df9998faff9245b80865b0ddf5958f71dc0874 WatchSource:0}: Error finding container 12ee10c97e4489286741ede947df9998faff9245b80865b0ddf5958f71dc0874: Status 404 returned error can't find the container with id 12ee10c97e4489286741ede947df9998faff9245b80865b0ddf5958f71dc0874 Oct 30 00:55:50.793905 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:50.793877 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx"] Oct 30 00:55:50.798169 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:50.798142 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1885f34e_6420_4b73_aab4_18d173a0f95a.slice/crio-cdddbf9cd832d01d7e3d8fa6bb00ffa9f93638a1831f71d72134a003b9cfe5b4 WatchSource:0}: Error finding container cdddbf9cd832d01d7e3d8fa6bb00ffa9f93638a1831f71d72134a003b9cfe5b4: Status 404 returned error can't find the container with id cdddbf9cd832d01d7e3d8fa6bb00ffa9f93638a1831f71d72134a003b9cfe5b4 Oct 30 00:55:51.084063 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.084028 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:51.240130 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.240088 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:51.550747 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.548288 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:51.625836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.625688 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" event={"ID":"25333518-39ac-4e6c-a406-80a09d216b14","Type":"ContainerStarted","Data":"5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767"} Oct 30 00:55:51.629138 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.626468 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" event={"ID":"25333518-39ac-4e6c-a406-80a09d216b14","Type":"ContainerStarted","Data":"12ee10c97e4489286741ede947df9998faff9245b80865b0ddf5958f71dc0874"} Oct 30 00:55:51.629138 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.627684 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:55:51.632727 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.631699 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" event={"ID":"1885f34e-6420-4b73-aab4-18d173a0f95a","Type":"ContainerStarted","Data":"6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c"} Oct 30 00:55:51.632727 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.631774 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" event={"ID":"1885f34e-6420-4b73-aab4-18d173a0f95a","Type":"ContainerStarted","Data":"cdddbf9cd832d01d7e3d8fa6bb00ffa9f93638a1831f71d72134a003b9cfe5b4"} Oct 30 00:55:51.632727 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.632521 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:55:51.674731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.668580 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" podStartSLOduration=2.668550581 podStartE2EDuration="2.668550581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:51.663556931 +0000 UTC m=+1289.575684073" watchObservedRunningTime="2026-10-30 00:55:51.668550581 +0000 UTC m=+1289.580677725" Oct 30 00:55:51.779043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:51.775699 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" podStartSLOduration=2.775676897 podStartE2EDuration="2.775676897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:51.704120496 +0000 UTC m=+1289.616247639" watchObservedRunningTime="2026-10-30 00:55:51.775676897 +0000 UTC m=+1289.687804038" Oct 30 00:55:52.841118 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:52.841083 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:53.105792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:53.102482 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:53.273342 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:53.273303 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.725173 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:54.725137 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.858235 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:54.858191 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.014874 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.014779 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.314731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.312339 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7"] Oct 30 00:55:55.340671 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.340642 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:55:55.366183 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.366157 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-610065-dockercfg-gbhqc\"" Oct 30 00:55:55.392195 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.392170 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-610065\"" Oct 30 00:55:55.543811 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.543698 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7"] Oct 30 00:55:55.590794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.585258 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzj4c\" (UniqueName: \"kubernetes.io/projected/9b39a48f-277c-477e-805b-4155d27e8b2b-kube-api-access-pzj4c\") pod \"payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7\" (UID: \"9b39a48f-277c-477e-805b-4155d27e8b2b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:55:55.590794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.585417 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b39a48f-277c-477e-805b-4155d27e8b2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7\" (UID: \"9b39a48f-277c-477e-805b-4155d27e8b2b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:55:55.596809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.596741 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v"] Oct 30 00:55:55.602383 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.602347 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:55:55.605016 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.604993 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v"] Oct 30 00:55:55.688524 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.688485 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b39a48f-277c-477e-805b-4155d27e8b2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7\" (UID: \"9b39a48f-277c-477e-805b-4155d27e8b2b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:55:55.688852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.688829 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-610065-997f67946-54k2v\" (UID: \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\") " pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:55:55.696208 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.696179 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gbp\" (UniqueName: \"kubernetes.io/projected/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-kube-api-access-n8gbp\") pod \"payload-processing-e2e-api-a-610065-997f67946-54k2v\" (UID: \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\") " pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:55:55.696642 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.696619 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzj4c\" (UniqueName: \"kubernetes.io/projected/9b39a48f-277c-477e-805b-4155d27e8b2b-kube-api-access-pzj4c\") pod \"payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7\" (UID: \"9b39a48f-277c-477e-805b-4155d27e8b2b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:55:55.744549 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.726283 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b39a48f-277c-477e-805b-4155d27e8b2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7\" (UID: \"9b39a48f-277c-477e-805b-4155d27e8b2b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:55:55.760402 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.760369 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzj4c\" (UniqueName: \"kubernetes.io/projected/9b39a48f-277c-477e-805b-4155d27e8b2b-kube-api-access-pzj4c\") pod \"payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7\" (UID: \"9b39a48f-277c-477e-805b-4155d27e8b2b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:55:55.803732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.803383 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-610065-997f67946-54k2v\" (UID: \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\") " pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:55:55.803732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.803429 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gbp\" (UniqueName: \"kubernetes.io/projected/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-kube-api-access-n8gbp\") pod \"payload-processing-e2e-api-a-610065-997f67946-54k2v\" (UID: \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\") " pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:55:55.825733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.804610 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-610065-997f67946-54k2v\" (UID: \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\") " pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:55:55.868663 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:55.868568 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gbp\" (UniqueName: \"kubernetes.io/projected/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-kube-api-access-n8gbp\") pod \"payload-processing-e2e-api-a-610065-997f67946-54k2v\" (UID: \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\") " pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:55:56.021848 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.021796 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:56.025218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.025181 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:56.048339 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.047721 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:55:56.058021 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.057748 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:55:56.078993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.078957 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:55:56.214823 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.207226 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:56.332652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.332184 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:56.633072 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:56.632388 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa413bc_463a_47e4_97d6_d6ec315ef3aa.slice/crio-46140d00826b7711ab4534e051abc715da38f8ad198555d5eadb5ce17d5663a8 WatchSource:0}: Error finding container 46140d00826b7711ab4534e051abc715da38f8ad198555d5eadb5ce17d5663a8: Status 404 returned error can't find the container with id 46140d00826b7711ab4534e051abc715da38f8ad198555d5eadb5ce17d5663a8 Oct 30 00:55:56.633197 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:56.633167 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b39a48f_277c_477e_805b_4155d27e8b2b.slice/crio-740a0f11d378571d7fc18039de26aa91a41fc22785acfb324dd01f9cb2c65c53 WatchSource:0}: Error finding container 740a0f11d378571d7fc18039de26aa91a41fc22785acfb324dd01f9cb2c65c53: Status 404 returned error can't find the container with id 740a0f11d378571d7fc18039de26aa91a41fc22785acfb324dd01f9cb2c65c53 Oct 30 00:55:56.635971 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.635950 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v"] Oct 30 00:55:56.641945 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.641919 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7"] Oct 30 00:55:56.755365 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.755334 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" event={"ID":"eaa413bc-463a-47e4-97d6-d6ec315ef3aa","Type":"ContainerStarted","Data":"46140d00826b7711ab4534e051abc715da38f8ad198555d5eadb5ce17d5663a8"} Oct 30 00:55:56.757644 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.757622 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" event={"ID":"9b39a48f-277c-477e-805b-4155d27e8b2b","Type":"ContainerStarted","Data":"740a0f11d378571d7fc18039de26aa91a41fc22785acfb324dd01f9cb2c65c53"} Oct 30 00:55:56.918729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:56.918448 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:57.797733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:57.797654 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" event={"ID":"eaa413bc-463a-47e4-97d6-d6ec315ef3aa","Type":"ContainerStarted","Data":"8d0d82bb99648dc4ab1e47a4aef592867109b930ecd4449ec8041218a53e10f0"} Oct 30 00:55:57.813738 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:57.811558 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:55:57.818862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:57.814269 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" event={"ID":"9b39a48f-277c-477e-805b-4155d27e8b2b","Type":"ContainerStarted","Data":"df9c55e8f75657fc394286f9934e4dd84eb8184641aa9cfb3af11481ff3094ea"} Oct 30 00:55:57.831769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:57.822283 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:55:57.856129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:57.856080 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" podStartSLOduration=2.8560648840000002 podStartE2EDuration="2.856064884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:57.853873351 +0000 UTC m=+1295.766000489" watchObservedRunningTime="2026-10-30 00:55:57.856064884 +0000 UTC m=+1295.768192022" Oct 30 00:55:57.877304 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:57.877200 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:57.938948 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:57.936726 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" podStartSLOduration=3.936684929 podStartE2EDuration="3.936684929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:57.893075438 +0000 UTC m=+1295.805202583" watchObservedRunningTime="2026-10-30 00:55:57.936684929 +0000 UTC m=+1295.848812071" Oct 30 00:55:58.111565 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.110980 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:58.121782 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.120607 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps"] Oct 30 00:55:58.134955 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.134919 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps"] Oct 30 00:55:58.135154 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.135061 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:55:58.146419 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.146384 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth"] Oct 30 00:55:58.174791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.151808 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-16c1bb\"" Oct 30 00:55:58.174791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.152012 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-16c1bb-dockercfg-4kfz6\"" Oct 30 00:55:58.185777 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.185724 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072afb9f-694b-40a8-8715-543942df57cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps\" (UID: \"072afb9f-694b-40a8-8715-543942df57cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:55:58.187729 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.186027 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7fxr\" (UniqueName: \"kubernetes.io/projected/072afb9f-694b-40a8-8715-543942df57cf-kube-api-access-c7fxr\") pod \"payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps\" (UID: \"072afb9f-694b-40a8-8715-543942df57cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:55:58.199733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.189015 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:55:58.251748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.246192 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth"] Oct 30 00:55:58.310735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.289813 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjlcd\" (UniqueName: \"kubernetes.io/projected/94d6be96-d442-4e00-b51c-6bf15b11a4b6-kube-api-access-jjlcd\") pod \"payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth\" (UID: \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:55:58.310735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.289859 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d6be96-d442-4e00-b51c-6bf15b11a4b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth\" (UID: \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:55:58.310735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.289916 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7fxr\" (UniqueName: \"kubernetes.io/projected/072afb9f-694b-40a8-8715-543942df57cf-kube-api-access-c7fxr\") pod \"payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps\" (UID: \"072afb9f-694b-40a8-8715-543942df57cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:55:58.310735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.290042 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072afb9f-694b-40a8-8715-543942df57cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps\" (UID: \"072afb9f-694b-40a8-8715-543942df57cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:55:58.310735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.291058 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072afb9f-694b-40a8-8715-543942df57cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps\" (UID: \"072afb9f-694b-40a8-8715-543942df57cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:55:58.467836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.412209 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjlcd\" (UniqueName: \"kubernetes.io/projected/94d6be96-d442-4e00-b51c-6bf15b11a4b6-kube-api-access-jjlcd\") pod \"payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth\" (UID: \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:55:58.467836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.412259 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d6be96-d442-4e00-b51c-6bf15b11a4b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth\" (UID: \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:55:58.477744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.476661 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d6be96-d442-4e00-b51c-6bf15b11a4b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth\" (UID: \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:55:58.536769 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.535942 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjlcd\" (UniqueName: \"kubernetes.io/projected/94d6be96-d442-4e00-b51c-6bf15b11a4b6-kube-api-access-jjlcd\") pod \"payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth\" (UID: \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:55:58.637730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.636831 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:55:58.722952 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.712512 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7fxr\" (UniqueName: \"kubernetes.io/projected/072afb9f-694b-40a8-8715-543942df57cf-kube-api-access-c7fxr\") pod \"payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps\" (UID: \"072afb9f-694b-40a8-8715-543942df57cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:55:58.903349 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:58.902092 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:55:59.039862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.038454 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:59.556051 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.556022 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps"] Oct 30 00:55:59.556592 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:55:59.556567 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072afb9f_694b_40a8_8715_543942df57cf.slice/crio-ab52654e1a1d4e7983d362e75eff0f6655ea2e8c50e0b9dab252e18aac34e500 WatchSource:0}: Error finding container ab52654e1a1d4e7983d362e75eff0f6655ea2e8c50e0b9dab252e18aac34e500: Status 404 returned error can't find the container with id ab52654e1a1d4e7983d362e75eff0f6655ea2e8c50e0b9dab252e18aac34e500 Oct 30 00:55:59.614866 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.613759 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth"] Oct 30 00:55:59.849459 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.849373 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" event={"ID":"94d6be96-d442-4e00-b51c-6bf15b11a4b6","Type":"ContainerStarted","Data":"205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171"} Oct 30 00:55:59.849459 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.849438 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" event={"ID":"94d6be96-d442-4e00-b51c-6bf15b11a4b6","Type":"ContainerStarted","Data":"b0f0252865071cf263363cffa8acb0452703af8cd5471de0511669707a364a9c"} Oct 30 00:55:59.849671 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.849556 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:55:59.851244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.851219 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" event={"ID":"072afb9f-694b-40a8-8715-543942df57cf","Type":"ContainerStarted","Data":"2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0"} Oct 30 00:55:59.851368 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.851264 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" event={"ID":"072afb9f-694b-40a8-8715-543942df57cf","Type":"ContainerStarted","Data":"ab52654e1a1d4e7983d362e75eff0f6655ea2e8c50e0b9dab252e18aac34e500"} Oct 30 00:55:59.851553 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.851537 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:55:59.904521 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.904448 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" podStartSLOduration=1.904433267 podStartE2EDuration="1.904433267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:59.902476767 +0000 UTC m=+1297.814603908" watchObservedRunningTime="2026-10-30 00:55:59.904433267 +0000 UTC m=+1297.816560421" Oct 30 00:55:59.940352 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:55:59.940289 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" podStartSLOduration=2.940270903 podStartE2EDuration="2.940270903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:59.937792366 +0000 UTC m=+1297.849919530" watchObservedRunningTime="2026-10-30 00:55:59.940270903 +0000 UTC m=+1297.852398046" Oct 30 00:56:00.632431 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:00.632388 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:00.808797 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:56:00.808744 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:56:02.126543 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.126506 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq"] Oct 30 00:56:02.130946 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.130925 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.169782 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.157901 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-353403-serving-cert\"" Oct 30 00:56:02.169782 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.158330 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq"] Oct 30 00:56:02.215534 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.214774 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:02.250959 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.250927 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ghcf\" (UniqueName: \"kubernetes.io/projected/72eec295-5b8b-42e8-9484-da7c6e892669-kube-api-access-4ghcf\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.251223 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.251200 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72eec295-5b8b-42e8-9484-da7c6e892669-openshift-service-ca\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.251324 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.251305 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-maas-api-tls\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.251429 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.251410 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-metrics-certs\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.342548 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.342452 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:02.353292 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.352531 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72eec295-5b8b-42e8-9484-da7c6e892669-openshift-service-ca\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.353292 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.352687 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-maas-api-tls\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.353292 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.352828 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-metrics-certs\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.353292 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.352889 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ghcf\" (UniqueName: \"kubernetes.io/projected/72eec295-5b8b-42e8-9484-da7c6e892669-kube-api-access-4ghcf\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.355396 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.353903 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72eec295-5b8b-42e8-9484-da7c6e892669-openshift-service-ca\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.367681 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.361373 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-maas-api-tls\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.367681 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.366571 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-metrics-certs\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.392132 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.391406 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ghcf\" (UniqueName: \"kubernetes.io/projected/72eec295-5b8b-42e8-9484-da7c6e892669-kube-api-access-4ghcf\") pod \"maas-api-e2e-api-a-353403-6f9bcf884b-78txq\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.462618 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.462585 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:02.463249 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.463202 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:02.807883 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.807466 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq"] Oct 30 00:56:02.810599 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:56:02.810570 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72eec295_5b8b_42e8_9484_da7c6e892669.slice/crio-ee21f2c4bd41eea8ce04ab417f0f025a5d46d8ba9b5aa4ebec613232064c2a03 WatchSource:0}: Error finding container ee21f2c4bd41eea8ce04ab417f0f025a5d46d8ba9b5aa4ebec613232064c2a03: Status 404 returned error can't find the container with id ee21f2c4bd41eea8ce04ab417f0f025a5d46d8ba9b5aa4ebec613232064c2a03 Oct 30 00:56:02.882732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:02.882414 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" event={"ID":"72eec295-5b8b-42e8-9484-da7c6e892669","Type":"ContainerStarted","Data":"ee21f2c4bd41eea8ce04ab417f0f025a5d46d8ba9b5aa4ebec613232064c2a03"} Oct 30 00:56:03.708074 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:03.708038 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 00:56:03.708538 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:03.708124 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 00:56:03.896554 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:03.896513 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" event={"ID":"72eec295-5b8b-42e8-9484-da7c6e892669","Type":"ContainerStarted","Data":"1b8f3a0a9cd509c140cd69109c7a3dd3af00a5e8d97985a54f6e187535149e3c"} Oct 30 00:56:03.897679 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:03.897654 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:03.940180 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:03.940103 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" podStartSLOduration=1.626547374 podStartE2EDuration="1.940079783s" totalImagesPullingTime="313.532409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:03.935071568 +0000 UTC m=+1301.847198710" watchObservedRunningTime="2026-10-30 00:56:03.940079783 +0000 UTC m=+1301.852206937" Oct 30 00:56:04.153380 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:04.153337 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.515151 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:04.514233 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.908881 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:04.908845 4517 generic.go:356] "Generic (PLEG): container finished" podID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerID="50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf" exitCode=0 Oct 30 00:56:04.909378 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:04.908923 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" event={"ID":"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018","Type":"ContainerDied","Data":"50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf"} Oct 30 00:56:04.909542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:04.909524 4517 scope.go:122] "RemoveContainer" containerID="50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf" Oct 30 00:56:04.911260 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:04.911233 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:56:04.911337 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:04.911301 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:56:04.911375 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:04.911335 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:56:05.281993 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:05.280556 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:05.403250 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:05.403216 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:05.919726 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:05.919674 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" event={"ID":"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018","Type":"ContainerStarted","Data":"660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3"} Oct 30 00:56:06.117040 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.117015 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:06.734450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.734415 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:56:06.735752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.735724 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:56:06.735890 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.735776 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 00:56:06.943367 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.943323 4517 generic.go:356] "Generic (PLEG): container finished" podID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerID="ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3" exitCode=0 Oct 30 00:56:06.948082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.944436 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" event={"ID":"dcae3bbb-d666-405f-8a3b-e7b2440711a8","Type":"ContainerDied","Data":"ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3"} Oct 30 00:56:06.948082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.944984 4517 scope.go:122] "RemoveContainer" containerID="ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3" Oct 30 00:56:06.948082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.947323 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:56:06.948082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.947420 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:56:06.948082 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:06.947473 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 00:56:07.520121 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:07.520071 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:56:07.736298 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:07.735450 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:56:07.736298 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:07.735519 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 00:56:07.957542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:07.957509 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" event={"ID":"dcae3bbb-d666-405f-8a3b-e7b2440711a8","Type":"ContainerStarted","Data":"de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32"} Oct 30 00:56:08.195041 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:08.194990 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:08.734501 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:08.734470 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:56:08.734758 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:08.734734 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 00:56:08.925030 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:08.924984 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:56:08.926243 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:08.926214 4517 patch_prober.go:29] interesting pod/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:56:08.926436 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:08.926415 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 00:56:09.292733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:09.291932 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:09.423779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:09.408559 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:09.735208 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:09.735127 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:56:09.735208 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:09.735182 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 00:56:09.798839 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:09.798800 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:56:09.854852 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:09.854821 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:56:09.855058 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:09.854889 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:56:09.925098 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:09.925064 4517 patch_prober.go:29] interesting pod/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:56:09.925358 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:09.925329 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 00:56:10.236452 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:10.236424 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:10.736221 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:10.735204 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:56:10.736221 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:10.735277 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 00:56:10.885737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:10.877885 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 00:56:10.915915 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:10.915883 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 00:56:10.924823 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:10.924790 4517 patch_prober.go:29] interesting pod/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:56:10.924989 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:10.924851 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 00:56:10.928361 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:10.928338 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:56:11.054409 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:11.054366 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:11.221735 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:11.193662 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:11.566736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:11.562027 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:11.874800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:11.867947 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:56:11.874800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:11.868027 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 00:56:11.955736 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:11.948456 4517 patch_prober.go:29] interesting pod/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:56:11.956590 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:11.948566 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 00:56:12.145904 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:12.145803 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:12.416806 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:12.415158 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:12.643914 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:12.643878 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:12.932740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:12.932144 4517 patch_prober.go:29] interesting pod/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:56:12.932740 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:12.932204 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 00:56:12.965347 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:12.965313 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:56:12.980121 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:12.980093 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 00:56:13.354573 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:13.353117 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:13.925093 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:13.925054 4517 patch_prober.go:29] interesting pod/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:56:13.925290 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:13.925132 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 00:56:14.227526 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:14.227424 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:14.788470 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:14.788439 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:14.929856 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:14.929826 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:56:14.931073 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:14.931053 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:56:16.079184 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:16.079151 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:16.378853 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:16.378761 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:16.495935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:16.495867 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:17.878367 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:17.878335 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j"] Oct 30 00:56:17.882917 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:17.882894 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:17.905778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:17.905744 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j"] Oct 30 00:56:17.909148 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:17.909106 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-62f5d1-serving-cert\"" Oct 30 00:56:17.987051 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:17.986985 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:18.000784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.000744 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.000981 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.000888 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9pn\" (UniqueName: \"kubernetes.io/projected/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-kube-api-access-8m9pn\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.000981 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.000933 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.001119 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.001101 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.101456 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.101424 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:18.102071 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.102047 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.102205 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.102144 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9pn\" (UniqueName: \"kubernetes.io/projected/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-kube-api-access-8m9pn\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.102205 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.102179 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.102357 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.102339 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.102925 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.102896 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.105164 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.105130 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.105262 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.105194 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.124724 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.124674 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9pn\" (UniqueName: \"kubernetes.io/projected/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-kube-api-access-8m9pn\") pod \"maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.197243 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.197151 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:18.222184 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.222101 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:18.740591 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:18.740553 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j"] Oct 30 00:56:18.757003 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:56:18.756957 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6bfebb_1bf7_4279_82c8_bd98ec90d134.slice/crio-66473d560dd8ae9018d4f9cee19573d32e2decdf8e81ee27f1599e2466b58bda WatchSource:0}: Error finding container 66473d560dd8ae9018d4f9cee19573d32e2decdf8e81ee27f1599e2466b58bda: Status 404 returned error can't find the container with id 66473d560dd8ae9018d4f9cee19573d32e2decdf8e81ee27f1599e2466b58bda Oct 30 00:56:19.212546 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:19.212499 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" event={"ID":"aa6bfebb-1bf7-4279-82c8-bd98ec90d134","Type":"ContainerStarted","Data":"f6ad25c0c492fc00ac2c0e617b8464c0ac0f0007a98e1f64b43b850a2e13c24e"} Oct 30 00:56:19.212942 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:19.212601 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:19.212942 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:19.212620 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" event={"ID":"aa6bfebb-1bf7-4279-82c8-bd98ec90d134","Type":"ContainerStarted","Data":"66473d560dd8ae9018d4f9cee19573d32e2decdf8e81ee27f1599e2466b58bda"} Oct 30 00:56:19.275481 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:19.257395 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" podStartSLOduration=1.923374372 podStartE2EDuration="2.257374694s" totalImagesPullingTime="334.000322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:19.2513347 +0000 UTC m=+1317.163461842" watchObservedRunningTime="2026-10-30 00:56:19.257374694 +0000 UTC m=+1317.169501835" Oct 30 00:56:20.062871 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:20.062837 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:21.837028 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:21.836990 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:22.027376 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:22.027344 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t"] Oct 30 00:56:22.037022 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:22.036985 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4127e2-wsq2t"] Oct 30 00:56:22.239326 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:22.239251 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0028d5f1-ba84-4517-981f-077ad1252946" path="/var/lib/kubelet/pods/0028d5f1-ba84-4517-981f-077ad1252946/volumes" Oct 30 00:56:23.335885 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.335858 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j"] Oct 30 00:56:23.341439 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.341419 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.365428 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.365402 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-610065-serving-cert\"" Oct 30 00:56:23.371552 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.371526 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j"] Oct 30 00:56:23.432002 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.431967 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:23.537603 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.537549 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-maas-api-tls\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.538144 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.538117 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba5971c-13b9-419e-af84-2d49cd38b2b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.538401 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.538375 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-metrics-certs\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.538783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.538694 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t889\" (UniqueName: \"kubernetes.io/projected/cba5971c-13b9-419e-af84-2d49cd38b2b5-kube-api-access-4t889\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.556819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.556784 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:23.640520 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.640421 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t889\" (UniqueName: \"kubernetes.io/projected/cba5971c-13b9-419e-af84-2d49cd38b2b5-kube-api-access-4t889\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.640520 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.640498 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-maas-api-tls\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.640820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.640615 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba5971c-13b9-419e-af84-2d49cd38b2b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.640820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.640772 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-metrics-certs\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.641515 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.641457 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba5971c-13b9-419e-af84-2d49cd38b2b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.643760 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.643735 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-metrics-certs\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.643839 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.643761 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-maas-api-tls\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.646784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.646764 4517 scope.go:122] "RemoveContainer" containerID="665c2d4c2c1d3fe390b1c63b382070125e45d41eedcff0f2240ccca654d15e7f" Oct 30 00:56:23.663258 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.663232 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t889\" (UniqueName: \"kubernetes.io/projected/cba5971c-13b9-419e-af84-2d49cd38b2b5-kube-api-access-4t889\") pod \"maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:23.667660 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.667639 4517 scope.go:122] "RemoveContainer" containerID="a8ae7f526f01762f0f4dd0fc15636151e0aa712768daca8964ae87abbd639b0e" Oct 30 00:56:23.954580 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:23.954489 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:24.276694 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:24.276504 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j"] Oct 30 00:56:24.279773 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:56:24.279528 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba5971c_13b9_419e_af84_2d49cd38b2b5.slice/crio-76f89cdc32df22a014f23f391f16142f6e7ea9f4ee0b7d11d390ef4e4bedcb49 WatchSource:0}: Error finding container 76f89cdc32df22a014f23f391f16142f6e7ea9f4ee0b7d11d390ef4e4bedcb49: Status 404 returned error can't find the container with id 76f89cdc32df22a014f23f391f16142f6e7ea9f4ee0b7d11d390ef4e4bedcb49 Oct 30 00:56:25.225991 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:25.225964 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 00:56:25.263809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:25.263771 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" event={"ID":"cba5971c-13b9-419e-af84-2d49cd38b2b5","Type":"ContainerStarted","Data":"17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37"} Oct 30 00:56:25.263987 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:25.263834 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" event={"ID":"cba5971c-13b9-419e-af84-2d49cd38b2b5","Type":"ContainerStarted","Data":"76f89cdc32df22a014f23f391f16142f6e7ea9f4ee0b7d11d390ef4e4bedcb49"} Oct 30 00:56:25.263987 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:25.263864 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:25.419270 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:25.419240 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:27.173382 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.173315 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" podStartSLOduration=3.747051695 podStartE2EDuration="4.173294661s" totalImagesPullingTime="426.242966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:25.308343532 +0000 UTC m=+1323.220470675" watchObservedRunningTime="2026-10-30 00:56:27.173294661 +0000 UTC m=+1325.085421803" Oct 30 00:56:27.177836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.177809 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz"] Oct 30 00:56:27.182582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.182563 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.198836 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.198813 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-16c1bb-serving-cert\"" Oct 30 00:56:27.212277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.212250 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz"] Oct 30 00:56:27.271069 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.271038 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:27.303748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.301836 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.303748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.301894 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.303748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.301962 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpd9\" (UniqueName: \"kubernetes.io/projected/04819ea4-d56b-455b-813a-dc677d867e3c-kube-api-access-hgpd9\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.303748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.302037 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04819ea4-d56b-455b-813a-dc677d867e3c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.403646 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.403612 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.403850 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.403664 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.403850 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.403736 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpd9\" (UniqueName: \"kubernetes.io/projected/04819ea4-d56b-455b-813a-dc677d867e3c-kube-api-access-hgpd9\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.403850 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.403795 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04819ea4-d56b-455b-813a-dc677d867e3c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.404582 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.404552 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04819ea4-d56b-455b-813a-dc677d867e3c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.405190 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.405158 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:27.406440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.406421 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.406781 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.406758 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.425901 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.425846 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpd9\" (UniqueName: \"kubernetes.io/projected/04819ea4-d56b-455b-813a-dc677d867e3c-kube-api-access-hgpd9\") pod \"maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.498009 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.497972 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:27.524223 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.524191 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:27.844377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:27.844350 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz"] Oct 30 00:56:27.845508 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:56:27.845482 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04819ea4_d56b_455b_813a_dc677d867e3c.slice/crio-8b9ff7b2564d929f30377dd2890a8183812ea42d35eae0e0094238df26e38f99 WatchSource:0}: Error finding container 8b9ff7b2564d929f30377dd2890a8183812ea42d35eae0e0094238df26e38f99: Status 404 returned error can't find the container with id 8b9ff7b2564d929f30377dd2890a8183812ea42d35eae0e0094238df26e38f99 Oct 30 00:56:28.297277 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:28.293884 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" event={"ID":"04819ea4-d56b-455b-813a-dc677d867e3c","Type":"ContainerStarted","Data":"8b9ff7b2564d929f30377dd2890a8183812ea42d35eae0e0094238df26e38f99"} Oct 30 00:56:29.309286 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:29.309254 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" event={"ID":"04819ea4-d56b-455b-813a-dc677d867e3c","Type":"ContainerStarted","Data":"91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698"} Oct 30 00:56:29.309820 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:29.309387 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:29.352755 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:29.350793 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" podStartSLOduration=1.9661099389999999 podStartE2EDuration="2.350777236s" totalImagesPullingTime="384.667297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:29.347371568 +0000 UTC m=+1327.259498708" watchObservedRunningTime="2026-10-30 00:56:29.350777236 +0000 UTC m=+1327.262904378" Oct 30 00:56:29.402046 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:29.402011 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:30.760176 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:56:30.760142 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:56:31.155059 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:31.155028 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:31.278506 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:31.278467 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:56:33.027106 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:33.027074 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:34.863416 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:34.863289 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:35.037443 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:35.037413 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4"] Oct 30 00:56:35.045329 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:35.045301 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4aec633-n5sf4"] Oct 30 00:56:35.322744 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:35.322683 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 00:56:36.244304 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:36.244262 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70a29be-83bc-4d79-8a25-92a23c802ebd" path="/var/lib/kubelet/pods/b70a29be-83bc-4d79-8a25-92a23c802ebd/volumes" Oct 30 00:56:37.310800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:37.310730 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:39.523945 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:39.523913 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:41.493334 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:41.493297 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:43.246414 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.246374 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8"] Oct 30 00:56:43.251034 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.251010 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.271864 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.271840 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a4fe0143-serving-cert\"" Oct 30 00:56:43.282342 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.282308 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8"] Oct 30 00:56:43.313028 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.312985 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-maas-api-tls\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.313178 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.313039 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ztff\" (UniqueName: \"kubernetes.io/projected/d9daf3da-99a9-4ab1-9def-c2fde73335b0-kube-api-access-5ztff\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.313221 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.313202 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9daf3da-99a9-4ab1-9def-c2fde73335b0-openshift-service-ca\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.313375 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.313359 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-metrics-certs\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.340312 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.340286 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:43.414876 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.414845 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9daf3da-99a9-4ab1-9def-c2fde73335b0-openshift-service-ca\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.415094 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.414980 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-metrics-certs\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.415686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.415177 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-maas-api-tls\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.415686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.415217 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ztff\" (UniqueName: \"kubernetes.io/projected/d9daf3da-99a9-4ab1-9def-c2fde73335b0-kube-api-access-5ztff\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.415686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.415639 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9daf3da-99a9-4ab1-9def-c2fde73335b0-openshift-service-ca\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.418318 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.418297 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-metrics-certs\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.418451 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.418333 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-maas-api-tls\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.438057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.438029 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ztff\" (UniqueName: \"kubernetes.io/projected/d9daf3da-99a9-4ab1-9def-c2fde73335b0-kube-api-access-5ztff\") pod \"maas-api-e2e-mt-a4fe0143-5b969575-4hdf8\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.468494 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.456480 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:43.568467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.568430 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:43.897418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:43.897388 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8"] Oct 30 00:56:43.899171 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:56:43.899142 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9daf3da_99a9_4ab1_9def_c2fde73335b0.slice/crio-5aa8eb521f1fde4f9b23f7aa5ada57316fad015d3713ba14b1b2e3d413248a40 WatchSource:0}: Error finding container 5aa8eb521f1fde4f9b23f7aa5ada57316fad015d3713ba14b1b2e3d413248a40: Status 404 returned error can't find the container with id 5aa8eb521f1fde4f9b23f7aa5ada57316fad015d3713ba14b1b2e3d413248a40 Oct 30 00:56:44.436148 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:44.436113 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" event={"ID":"d9daf3da-99a9-4ab1-9def-c2fde73335b0","Type":"ContainerStarted","Data":"8f2e75a1585002154987b4eb05579e48cd403a8db91f315e63e8eb80b8fe8bef"} Oct 30 00:56:44.436785 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:44.436768 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:44.437191 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:44.437169 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" event={"ID":"d9daf3da-99a9-4ab1-9def-c2fde73335b0","Type":"ContainerStarted","Data":"5aa8eb521f1fde4f9b23f7aa5ada57316fad015d3713ba14b1b2e3d413248a40"} Oct 30 00:56:44.480391 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:44.480328 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" podStartSLOduration=1.066976973 podStartE2EDuration="1.480312256s" totalImagesPullingTime="413.335283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:44.475610671 +0000 UTC m=+1342.387737836" watchObservedRunningTime="2026-10-30 00:56:44.480312256 +0000 UTC m=+1342.392439396" Oct 30 00:56:44.896737 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:44.890407 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:45.339174 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:45.339140 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:45.559101 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:45.559070 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:46.028727 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:46.028399 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:46.636011 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:46.635970 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:47.204963 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:47.204905 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:49.177211 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:49.177168 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:50.075504 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:50.075466 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:50.201184 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:50.201113 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:51.125801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:51.125767 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:51.484093 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:51.484019 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:56:51.861717 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:51.861677 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:51.982492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:51.982458 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:53.299594 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:53.299531 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:53.424059 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:53.421344 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:53.546916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:53.546887 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:55.212842 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:55.212807 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:56.926957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:56.926917 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:59.160205 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:59.160168 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:59.321123 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:59.321077 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5"] Oct 30 00:56:59.494399 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:59.494315 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:59.615278 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:56:59.615230 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:00.781418 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:57:00.781383 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:57:01.323067 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:01.323027 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:03.029020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:03.028988 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh"] Oct 30 00:57:03.039354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:03.039324 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7fe2f25-bxpfh"] Oct 30 00:57:03.064440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:03.064408 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:04.237439 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:04.237405 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1" path="/var/lib/kubelet/pods/9dd3ddd5-5e59-44e4-8254-fbb55fa66dd1/volumes" Oct 30 00:57:04.321402 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:04.321348 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" podUID="355623bc-0b4e-46e2-9077-6ea574101991" containerName="payload-processing" containerID="cri-o://66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c" gracePeriod=25 Oct 30 00:57:04.321645 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:04.321561 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:04.909335 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:04.909294 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:05.781537 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:05.781497 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:05.906239 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:05.906188 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:06.028302 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:06.028265 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:07.045486 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:07.045452 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:07.183866 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:07.183807 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:07.305784 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:07.305631 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:09.338231 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:09.338195 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:09.475353 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:09.475320 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:10.132748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:10.132675 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:10.612944 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:10.612913 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:11.100127 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:11.100093 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:11.824728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:11.824674 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:12.708181 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:12.708143 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:13.962325 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:13.962293 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:14.111854 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:14.111810 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" podUID="355623bc-0b4e-46e2-9077-6ea574101991" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:57:14.143262 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:14.143211 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:14.284255 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:14.284154 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:14.429731 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:14.429674 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:14.559843 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:14.559741 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:14.918542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:14.918454 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:16.683595 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:16.683561 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:17.094556 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:17.094522 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:17.231195 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:17.231155 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:18.040494 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:18.040452 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc"] Oct 30 00:57:18.049652 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:18.049620 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dc0096-crmgc"] Oct 30 00:57:18.237476 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:18.237441 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b" path="/var/lib/kubelet/pods/70bdd1c7-1d5f-4a00-8bd7-e2db7ac54a9b/volumes" Oct 30 00:57:19.106325 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:19.106289 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:19.219417 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:19.219388 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:21.385005 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:21.384967 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:23.432115 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:23.432071 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:23.718093 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:23.718016 4517 scope.go:122] "RemoveContainer" containerID="70f140750f4f67005b50658b01d2ec98ce279702e3a1aeab3111ac71aaf78322" Oct 30 00:57:24.082189 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.082139 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" podUID="355623bc-0b4e-46e2-9077-6ea574101991" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:57:24.635339 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.635316 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:57:24.717458 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.717421 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355623bc-0b4e-46e2-9077-6ea574101991-kube-api-access-sx8q5\" (UniqueName: \"kubernetes.io/projected/355623bc-0b4e-46e2-9077-6ea574101991-kube-api-access-sx8q5\") pod \"355623bc-0b4e-46e2-9077-6ea574101991\" (UID: \"355623bc-0b4e-46e2-9077-6ea574101991\") " Oct 30 00:57:24.717642 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.717625 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355623bc-0b4e-46e2-9077-6ea574101991-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355623bc-0b4e-46e2-9077-6ea574101991-plugins-config-volume\") pod \"355623bc-0b4e-46e2-9077-6ea574101991\" (UID: \"355623bc-0b4e-46e2-9077-6ea574101991\") " Oct 30 00:57:24.718058 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.718032 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355623bc-0b4e-46e2-9077-6ea574101991-plugins-config-volume" pod "355623bc-0b4e-46e2-9077-6ea574101991" (UID: "355623bc-0b4e-46e2-9077-6ea574101991"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:24.718488 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.718465 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355623bc-0b4e-46e2-9077-6ea574101991-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:57:24.719975 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.719950 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355623bc-0b4e-46e2-9077-6ea574101991-kube-api-access-sx8q5" pod "355623bc-0b4e-46e2-9077-6ea574101991" (UID: "355623bc-0b4e-46e2-9077-6ea574101991"). InnerVolumeSpecName "kube-api-access-sx8q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:24.819940 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.819900 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8q5\" (UniqueName: \"kubernetes.io/projected/355623bc-0b4e-46e2-9077-6ea574101991-kube-api-access-sx8q5\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:57:24.935570 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.935466 4517 generic.go:356] "Generic (PLEG): container finished" podID="355623bc-0b4e-46e2-9077-6ea574101991" containerID="66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c" exitCode=0 Oct 30 00:57:24.935570 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.935512 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" Oct 30 00:57:24.935862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.935561 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" event={"ID":"355623bc-0b4e-46e2-9077-6ea574101991","Type":"ContainerDied","Data":"66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c"} Oct 30 00:57:24.935862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.935620 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5" event={"ID":"355623bc-0b4e-46e2-9077-6ea574101991","Type":"ContainerDied","Data":"f652a2dcfce7272740118546e692a843ee95a8fa7070368b17807813a2ae95c9"} Oct 30 00:57:24.935862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.935648 4517 scope.go:122] "RemoveContainer" containerID="66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c" Oct 30 00:57:24.948055 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.948035 4517 scope.go:122] "RemoveContainer" containerID="66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c" Oct 30 00:57:24.948407 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:57:24.948388 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c\": container with ID starting with 66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c not found: ID does not exist" containerID="66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c" Oct 30 00:57:24.948467 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.948418 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c"} err="failed to get container status \"66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c\": rpc error: code = NotFound desc = could not find container \"66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c\": container with ID starting with 66b499ee0bd900f5252cb78b5e1a2a1d66a4062c01415f791fc6d6bcde84cb3c not found: ID does not exist" Oct 30 00:57:24.989131 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:24.989088 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5"] Oct 30 00:57:25.022751 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:25.018061 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:25.022751 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:25.019441 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-6mxk5"] Oct 30 00:57:25.340028 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:25.339990 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:25.492233 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:25.492199 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:25.926942 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:25.926914 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:26.236761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:26.236681 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355623bc-0b4e-46e2-9077-6ea574101991" path="/var/lib/kubelet/pods/355623bc-0b4e-46e2-9077-6ea574101991/volumes" Oct 30 00:57:26.367265 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:26.367223 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:26.854676 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:26.854645 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:26.981765 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:26.981729 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:27.332153 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:27.332111 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:27.451967 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:27.451635 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:29.317620 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:29.317568 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:29.444788 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:29.444758 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:29.560564 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:29.560530 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:30.776683 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:57:30.776648 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:57:31.218319 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:31.218271 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:32.849955 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:32.849874 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:33.044067 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:33.044028 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst"] Oct 30 00:57:33.055833 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:33.055792 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b18da0-vqqst"] Oct 30 00:57:33.555862 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:33.555830 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:34.237835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:34.237803 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3dcf77-6987-443d-bd45-ba998e1f2cc6" path="/var/lib/kubelet/pods/fd3dcf77-6987-443d-bd45-ba998e1f2cc6/volumes" Oct 30 00:57:34.698436 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:34.698405 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:36.727320 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:36.727205 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:38.174057 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:38.174022 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:38.649377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:38.649340 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:40.526514 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:40.526476 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:42.032727 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:42.032677 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph"] Oct 30 00:57:42.042482 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:42.042454 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0591d4-x74ph"] Oct 30 00:57:42.238663 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:42.238618 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9096fe6-7e29-418d-97be-ded94d123693" path="/var/lib/kubelet/pods/b9096fe6-7e29-418d-97be-ded94d123693/volumes" Oct 30 00:57:43.460526 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:43.460492 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:44.143724 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:44.143686 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:45.842995 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:45.842945 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:46.637175 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:46.637143 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:47.690312 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:47.690271 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:49.366589 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:49.366556 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:51.077637 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:51.077605 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7"] Oct 30 00:57:51.128167 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:51.128134 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:51.247393 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:51.247361 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:53.212301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:53.212263 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:53.896800 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:53.896764 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:54.024693 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:54.024628 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:54.141020 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:54.140989 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:56.024829 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:56.024783 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:56.082819 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:56.082767 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" podUID="3b9611dd-f57d-407d-a795-ca22244f4ef6" containerName="payload-pre-processing" containerID="cri-o://8bb71df0ac6fbd7e24e613d8d0e25925d05d8c2996d914e5f08b3775cb156737" gracePeriod=25 Oct 30 00:57:56.419092 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:56.419050 4517 generic.go:356] "Generic (PLEG): container finished" podID="3b9611dd-f57d-407d-a795-ca22244f4ef6" containerID="8bb71df0ac6fbd7e24e613d8d0e25925d05d8c2996d914e5f08b3775cb156737" exitCode=0 Oct 30 00:57:56.419409 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:56.419385 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" event={"ID":"3b9611dd-f57d-407d-a795-ca22244f4ef6","Type":"ContainerDied","Data":"8bb71df0ac6fbd7e24e613d8d0e25925d05d8c2996d914e5f08b3775cb156737"} Oct 30 00:57:56.547249 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:56.547207 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:56.731058 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:56.731022 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:56.865803 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:56.865668 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:56.987933 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:56.987902 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:57:57.036732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.034910 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b9611dd-f57d-407d-a795-ca22244f4ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9611dd-f57d-407d-a795-ca22244f4ef6-plugins-config-volume\") pod \"3b9611dd-f57d-407d-a795-ca22244f4ef6\" (UID: \"3b9611dd-f57d-407d-a795-ca22244f4ef6\") " Oct 30 00:57:57.036732 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.035433 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9611dd-f57d-407d-a795-ca22244f4ef6-plugins-config-volume" pod "3b9611dd-f57d-407d-a795-ca22244f4ef6" (UID: "3b9611dd-f57d-407d-a795-ca22244f4ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:57.046664 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.037525 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9611dd-f57d-407d-a795-ca22244f4ef6-kube-api-access-7tg2q\" (UniqueName: \"kubernetes.io/projected/3b9611dd-f57d-407d-a795-ca22244f4ef6-kube-api-access-7tg2q\") pod \"3b9611dd-f57d-407d-a795-ca22244f4ef6\" (UID: \"3b9611dd-f57d-407d-a795-ca22244f4ef6\") " Oct 30 00:57:57.046664 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.038489 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9611dd-f57d-407d-a795-ca22244f4ef6-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.049960 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.049930 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9611dd-f57d-407d-a795-ca22244f4ef6-kube-api-access-7tg2q" pod "3b9611dd-f57d-407d-a795-ca22244f4ef6" (UID: "3b9611dd-f57d-407d-a795-ca22244f4ef6"). InnerVolumeSpecName "kube-api-access-7tg2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:57.141648 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.141559 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tg2q\" (UniqueName: \"kubernetes.io/projected/3b9611dd-f57d-407d-a795-ca22244f4ef6-kube-api-access-7tg2q\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.430916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.430823 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" event={"ID":"3b9611dd-f57d-407d-a795-ca22244f4ef6","Type":"ContainerDied","Data":"125737ea0bbbe750a04645e4c0ebc03b624eaeb811ef79dc88d7fc138780080b"} Oct 30 00:57:57.430916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.430885 4517 scope.go:122] "RemoveContainer" containerID="8bb71df0ac6fbd7e24e613d8d0e25925d05d8c2996d914e5f08b3775cb156737" Oct 30 00:57:57.430916 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.430842 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7" Oct 30 00:57:57.506754 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.506682 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7"] Oct 30 00:57:57.512443 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:57.512414 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6867b8-d5f6c9b75-j7xm7"] Oct 30 00:57:58.237936 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:58.237898 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9611dd-f57d-407d-a795-ca22244f4ef6" path="/var/lib/kubelet/pods/3b9611dd-f57d-407d-a795-ca22244f4ef6/volumes" Oct 30 00:57:58.834084 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:57:58.834045 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:00.761024 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:58:00.760988 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:58:00.989183 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:00.989146 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:02.611111 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:02.611076 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:03.349439 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:03.349409 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:03.475990 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:03.475949 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:03.602036 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:03.601930 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:06.086752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:06.082421 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:07.774558 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:07.774520 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:07.896752 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:07.896668 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:09.716472 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:09.716440 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:11.112902 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:11.112859 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:11.222912 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:11.222872 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:11.611178 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:11.611143 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:12.302778 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.301942 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:12.422039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.422006 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:12.453796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.453763 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw"] Oct 30 00:58:12.454474 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.454458 4517 cpu_manager.go:414] "containerMap: removing container" podUID="3b9611dd-f57d-407d-a795-ca22244f4ef6" containerName="payload-pre-processing" Oct 30 00:58:12.454540 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.454476 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9611dd-f57d-407d-a795-ca22244f4ef6" containerName="payload-pre-processing" Oct 30 00:58:12.454540 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.454500 4517 cpu_manager.go:414] "containerMap: removing container" podUID="355623bc-0b4e-46e2-9077-6ea574101991" containerName="payload-processing" Oct 30 00:58:12.454540 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.454505 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355623bc-0b4e-46e2-9077-6ea574101991" containerName="payload-processing" Oct 30 00:58:12.454635 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.454607 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9611dd-f57d-407d-a795-ca22244f4ef6" containerName="payload-pre-processing" Oct 30 00:58:12.454635 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.454620 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="355623bc-0b4e-46e2-9077-6ea574101991" containerName="payload-processing" Oct 30 00:58:12.458248 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.458226 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:12.475179 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.475141 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d8090513\"" Oct 30 00:58:12.475823 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.475780 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d8090513-dockercfg-4mf4w\"" Oct 30 00:58:12.477596 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.477574 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw"] Oct 30 00:58:12.530284 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.530238 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd797a2-2e92-4084-b06a-1318ebd694cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw\" (UID: \"1cd797a2-2e92-4084-b06a-1318ebd694cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:12.530780 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.530651 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dms\" (UniqueName: \"kubernetes.io/projected/1cd797a2-2e92-4084-b06a-1318ebd694cb-kube-api-access-b6dms\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw\" (UID: \"1cd797a2-2e92-4084-b06a-1318ebd694cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:12.790875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.774137 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw"] Oct 30 00:58:12.790875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.779399 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dms\" (UniqueName: \"kubernetes.io/projected/1cd797a2-2e92-4084-b06a-1318ebd694cb-kube-api-access-b6dms\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw\" (UID: \"1cd797a2-2e92-4084-b06a-1318ebd694cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:12.790875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.779618 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd797a2-2e92-4084-b06a-1318ebd694cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw\" (UID: \"1cd797a2-2e92-4084-b06a-1318ebd694cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:12.790875 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.780680 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd797a2-2e92-4084-b06a-1318ebd694cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw\" (UID: \"1cd797a2-2e92-4084-b06a-1318ebd694cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:12.799457 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.797036 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:12.809266 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.809010 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw"] Oct 30 00:58:12.880828 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.879634 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dms\" (UniqueName: \"kubernetes.io/projected/1cd797a2-2e92-4084-b06a-1318ebd694cb-kube-api-access-b6dms\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw\" (UID: \"1cd797a2-2e92-4084-b06a-1318ebd694cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:12.882502 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.882259 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zkl\" (UniqueName: \"kubernetes.io/projected/3f998c5a-b2c5-4f94-81ff-875400c91125-kube-api-access-f7zkl\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw\" (UID: \"3f998c5a-b2c5-4f94-81ff-875400c91125\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:12.882809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.882785 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f998c5a-b2c5-4f94-81ff-875400c91125-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw\" (UID: \"3f998c5a-b2c5-4f94-81ff-875400c91125\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:12.986495 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.986278 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zkl\" (UniqueName: \"kubernetes.io/projected/3f998c5a-b2c5-4f94-81ff-875400c91125-kube-api-access-f7zkl\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw\" (UID: \"3f998c5a-b2c5-4f94-81ff-875400c91125\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:12.986495 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.986449 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f998c5a-b2c5-4f94-81ff-875400c91125-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw\" (UID: \"3f998c5a-b2c5-4f94-81ff-875400c91125\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:12.988420 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:12.988383 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f998c5a-b2c5-4f94-81ff-875400c91125-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw\" (UID: \"3f998c5a-b2c5-4f94-81ff-875400c91125\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:13.010796 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.010762 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zkl\" (UniqueName: \"kubernetes.io/projected/3f998c5a-b2c5-4f94-81ff-875400c91125-kube-api-access-f7zkl\") pod \"payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw\" (UID: \"3f998c5a-b2c5-4f94-81ff-875400c91125\") " pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:13.074088 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.073979 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:13.171748 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.171701 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:13.391346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.391317 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw"] Oct 30 00:58:13.393102 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:58:13.393058 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd797a2_2e92_4084_b06a_1318ebd694cb.slice/crio-0230584cebf7743cf7e03d9e6d4c90071a2401eb9294ae3b8b24a9c3cf992e07 WatchSource:0}: Error finding container 0230584cebf7743cf7e03d9e6d4c90071a2401eb9294ae3b8b24a9c3cf992e07: Status 404 returned error can't find the container with id 0230584cebf7743cf7e03d9e6d4c90071a2401eb9294ae3b8b24a9c3cf992e07 Oct 30 00:58:13.492720 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.492677 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw"] Oct 30 00:58:13.494655 ip-10-0-132-84 kubenswrapper[4517]: W1030 00:58:13.494632 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f998c5a_b2c5_4f94_81ff_875400c91125.slice/crio-fe04a0bb6fb02a140d92285cad99f2a44ecfa41041d40567d3e8fabb652b94be WatchSource:0}: Error finding container fe04a0bb6fb02a140d92285cad99f2a44ecfa41041d40567d3e8fabb652b94be: Status 404 returned error can't find the container with id fe04a0bb6fb02a140d92285cad99f2a44ecfa41041d40567d3e8fabb652b94be Oct 30 00:58:13.884991 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.884950 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" event={"ID":"3f998c5a-b2c5-4f94-81ff-875400c91125","Type":"ContainerStarted","Data":"f5d0a36ef67e29743aacde6b722af0bde317675bcbc9720d848ea262c1cf61e0"} Oct 30 00:58:13.885174 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.885026 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:13.885174 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.885051 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" event={"ID":"3f998c5a-b2c5-4f94-81ff-875400c91125","Type":"ContainerStarted","Data":"fe04a0bb6fb02a140d92285cad99f2a44ecfa41041d40567d3e8fabb652b94be"} Oct 30 00:58:13.886943 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.886913 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" event={"ID":"1cd797a2-2e92-4084-b06a-1318ebd694cb","Type":"ContainerStarted","Data":"c8c78df6c4c2d9b0230c288cbde03446ca5f5f8b31a406c889e28a913509343e"} Oct 30 00:58:13.887111 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.886967 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" event={"ID":"1cd797a2-2e92-4084-b06a-1318ebd694cb","Type":"ContainerStarted","Data":"0230584cebf7743cf7e03d9e6d4c90071a2401eb9294ae3b8b24a9c3cf992e07"} Oct 30 00:58:13.887111 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.887051 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:13.922805 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.922756 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" podStartSLOduration=1.922741635 podStartE2EDuration="1.922741635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:13.922382278 +0000 UTC m=+1431.834509419" watchObservedRunningTime="2026-10-30 00:58:13.922741635 +0000 UTC m=+1431.834868772" Oct 30 00:58:13.928783 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.928758 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:13.961637 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:13.961590 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" podStartSLOduration=1.961576598 podStartE2EDuration="1.961576598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:13.958684245 +0000 UTC m=+1431.870811386" watchObservedRunningTime="2026-10-30 00:58:13.961576598 +0000 UTC m=+1431.873703791" Oct 30 00:58:15.034934 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:15.034905 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:15.769418 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:15.769383 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:17.620399 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:17.620364 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:19.532394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:19.532361 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:21.527051 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:21.527014 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:22.715951 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:22.715915 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:22.839041 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:22.838978 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:23.136962 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:23.136927 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:23.746136 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:23.746108 4517 scope.go:122] "RemoveContainer" containerID="c296bccca6c6e63f72fff4149b2de063da195001aad0def8d646dc36c9ec7858" Oct 30 00:58:23.756166 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:23.756144 4517 scope.go:122] "RemoveContainer" containerID="f514650bf89a5f8ca7e76ca254edb6251e0a5bebe374a34199180624a0b3e339" Oct 30 00:58:23.767289 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:23.767265 4517 scope.go:122] "RemoveContainer" containerID="3efb3cc29accda34a4f57312da7ba59bc9165e3cae96ece9f8b3b3db6d603313" Oct 30 00:58:24.809955 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:24.809919 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:24.898233 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:24.898201 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:58:24.898439 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:24.898282 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:58:26.598127 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:26.598093 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:28.213785 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:28.213755 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:29.828257 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:29.828225 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:30.760580 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:58:30.760529 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:58:31.568926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:31.568873 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:33.128613 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:33.128578 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:34.856183 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:34.856150 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:36.215816 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:36.215696 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:36.928007 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:36.927961 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:38.829230 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:38.829192 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:40.909766 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:40.909686 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:41.041155 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:41.041081 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:41.164050 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:41.163952 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:42.450069 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:42.450036 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n"] Oct 30 00:58:42.450642 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:42.450280 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" podUID="71412d84-5e97-4794-aea9-4418023285d0" containerName="maas-api" containerID="cri-o://fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6" gracePeriod=30 Oct 30 00:58:42.528021 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:42.527993 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:43.009227 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.009191 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:58:43.056935 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.056895 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-maas-api-tls\") pod \"71412d84-5e97-4794-aea9-4418023285d0\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " Oct 30 00:58:43.057163 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.057025 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71412d84-5e97-4794-aea9-4418023285d0-kube-api-access-rsmwh\" (UniqueName: \"kubernetes.io/projected/71412d84-5e97-4794-aea9-4418023285d0-kube-api-access-rsmwh\") pod \"71412d84-5e97-4794-aea9-4418023285d0\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " Oct 30 00:58:43.057244 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.057197 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71412d84-5e97-4794-aea9-4418023285d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71412d84-5e97-4794-aea9-4418023285d0-openshift-service-ca\") pod \"71412d84-5e97-4794-aea9-4418023285d0\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " Oct 30 00:58:43.057321 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.057264 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-metrics-certs\") pod \"71412d84-5e97-4794-aea9-4418023285d0\" (UID: \"71412d84-5e97-4794-aea9-4418023285d0\") " Oct 30 00:58:43.061094 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.059592 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71412d84-5e97-4794-aea9-4418023285d0-openshift-service-ca" pod "71412d84-5e97-4794-aea9-4418023285d0" (UID: "71412d84-5e97-4794-aea9-4418023285d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:43.097196 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.097103 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71412d84-5e97-4794-aea9-4418023285d0-kube-api-access-rsmwh" pod "71412d84-5e97-4794-aea9-4418023285d0" (UID: "71412d84-5e97-4794-aea9-4418023285d0"). InnerVolumeSpecName "kube-api-access-rsmwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:43.113564 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.111316 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-metrics-certs" pod "71412d84-5e97-4794-aea9-4418023285d0" (UID: "71412d84-5e97-4794-aea9-4418023285d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:43.116739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.116652 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-maas-api-tls" pod "71412d84-5e97-4794-aea9-4418023285d0" (UID: "71412d84-5e97-4794-aea9-4418023285d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:43.184723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.184102 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:43.184723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.184137 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsmwh\" (UniqueName: \"kubernetes.io/projected/71412d84-5e97-4794-aea9-4418023285d0-kube-api-access-rsmwh\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:43.184723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.184152 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71412d84-5e97-4794-aea9-4418023285d0-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:43.184723 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.184166 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71412d84-5e97-4794-aea9-4418023285d0-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:43.254886 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.254839 4517 generic.go:356] "Generic (PLEG): container finished" podID="71412d84-5e97-4794-aea9-4418023285d0" containerID="fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6" exitCode=0 Oct 30 00:58:43.255094 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.255049 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" event={"ID":"71412d84-5e97-4794-aea9-4418023285d0","Type":"ContainerDied","Data":"fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6"} Oct 30 00:58:43.255170 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.255094 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" event={"ID":"71412d84-5e97-4794-aea9-4418023285d0","Type":"ContainerDied","Data":"503f78ca3c9a33fbb8e1f11b4bc716e53a1dd3e93a682f3aa3aeb614f2e52c92"} Oct 30 00:58:43.255170 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.255125 4517 scope.go:122] "RemoveContainer" containerID="fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6" Oct 30 00:58:43.255367 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.255283 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n" Oct 30 00:58:43.257926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.256321 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:43.323472 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.323126 4517 scope.go:122] "RemoveContainer" containerID="fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6" Oct 30 00:58:43.323979 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:58:43.323692 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6\": container with ID starting with fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6 not found: ID does not exist" containerID="fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6" Oct 30 00:58:43.323979 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.323908 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6"} err="failed to get container status \"fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6\": rpc error: code = NotFound desc = could not find container \"fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6\": container with ID starting with fe57d83be1949b647ada6438bfd65536c59941bb60980f9c7676de79daa18ef6 not found: ID does not exist" Oct 30 00:58:43.363828 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.363794 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n"] Oct 30 00:58:43.367791 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.367765 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6867b8-5d5748f499-xxm6n"] Oct 30 00:58:43.391216 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.391186 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:43.521589 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.521513 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:43.976320 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:43.976282 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:44.238346 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:44.238263 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71412d84-5e97-4794-aea9-4418023285d0" path="/var/lib/kubelet/pods/71412d84-5e97-4794-aea9-4418023285d0/volumes" Oct 30 00:58:45.282026 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:45.281993 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:47.047280 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:47.047248 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:48.130717 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:48.130669 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v"] Oct 30 00:58:48.170276 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:48.170239 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7"] Oct 30 00:58:48.382394 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:48.381190 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:48.495837 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:48.495797 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:49.217785 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:49.212701 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:51.030679 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:51.030642 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:51.160919 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:51.160858 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:51.473681 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:51.473642 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd"] Oct 30 00:58:51.474513 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:51.474481 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" containerID="cri-o://de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32" gracePeriod=30 Oct 30 00:58:51.536420 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:51.536385 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:53.108885 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.108854 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:53.131483 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.131451 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" podUID="eaa413bc-463a-47e4-97d6-d6ec315ef3aa" containerName="payload-processing" containerID="cri-o://8d0d82bb99648dc4ab1e47a4aef592867109b930ecd4449ec8041218a53e10f0" gracePeriod=25 Oct 30 00:58:53.173193 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.173152 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" podUID="9b39a48f-277c-477e-805b-4155d27e8b2b" containerName="payload-pre-processing" containerID="cri-o://df9c55e8f75657fc394286f9934e4dd84eb8184641aa9cfb3af11481ff3094ea" gracePeriod=25 Oct 30 00:58:53.334835 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.334743 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:53.345110 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.345081 4517 generic.go:356] "Generic (PLEG): container finished" podID="eaa413bc-463a-47e4-97d6-d6ec315ef3aa" containerID="8d0d82bb99648dc4ab1e47a4aef592867109b930ecd4449ec8041218a53e10f0" exitCode=0 Oct 30 00:58:53.345262 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.345212 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" event={"ID":"eaa413bc-463a-47e4-97d6-d6ec315ef3aa","Type":"ContainerDied","Data":"8d0d82bb99648dc4ab1e47a4aef592867109b930ecd4449ec8041218a53e10f0"} Oct 30 00:58:53.346918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.346892 4517 generic.go:356] "Generic (PLEG): container finished" podID="9b39a48f-277c-477e-805b-4155d27e8b2b" containerID="df9c55e8f75657fc394286f9934e4dd84eb8184641aa9cfb3af11481ff3094ea" exitCode=0 Oct 30 00:58:53.347054 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.346999 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" event={"ID":"9b39a48f-277c-477e-805b-4155d27e8b2b","Type":"ContainerDied","Data":"df9c55e8f75657fc394286f9934e4dd84eb8184641aa9cfb3af11481ff3094ea"} Oct 30 00:58:53.553613 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.553584 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:58:53.631420 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.631384 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:58:53.732228 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.732136 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b39a48f-277c-477e-805b-4155d27e8b2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b39a48f-277c-477e-805b-4155d27e8b2b-plugins-config-volume\") pod \"9b39a48f-277c-477e-805b-4155d27e8b2b\" (UID: \"9b39a48f-277c-477e-805b-4155d27e8b2b\") " Oct 30 00:58:53.732400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.732237 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-plugins-config-volume\") pod \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\" (UID: \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\") " Oct 30 00:58:53.732400 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.732306 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-kube-api-access-n8gbp\" (UniqueName: \"kubernetes.io/projected/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-kube-api-access-n8gbp\") pod \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\" (UID: \"eaa413bc-463a-47e4-97d6-d6ec315ef3aa\") " Oct 30 00:58:53.732549 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.732437 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b39a48f-277c-477e-805b-4155d27e8b2b-kube-api-access-pzj4c\" (UniqueName: \"kubernetes.io/projected/9b39a48f-277c-477e-805b-4155d27e8b2b-kube-api-access-pzj4c\") pod \"9b39a48f-277c-477e-805b-4155d27e8b2b\" (UID: \"9b39a48f-277c-477e-805b-4155d27e8b2b\") " Oct 30 00:58:53.732609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.732555 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b39a48f-277c-477e-805b-4155d27e8b2b-plugins-config-volume" pod "9b39a48f-277c-477e-805b-4155d27e8b2b" (UID: "9b39a48f-277c-477e-805b-4155d27e8b2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:53.732691 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.732666 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-plugins-config-volume" pod "eaa413bc-463a-47e4-97d6-d6ec315ef3aa" (UID: "eaa413bc-463a-47e4-97d6-d6ec315ef3aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:53.733345 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.733320 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b39a48f-277c-477e-805b-4155d27e8b2b-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:53.733459 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.733350 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:53.734989 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.734961 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-kube-api-access-n8gbp" pod "eaa413bc-463a-47e4-97d6-d6ec315ef3aa" (UID: "eaa413bc-463a-47e4-97d6-d6ec315ef3aa"). InnerVolumeSpecName "kube-api-access-n8gbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:53.735089 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.734998 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b39a48f-277c-477e-805b-4155d27e8b2b-kube-api-access-pzj4c" pod "9b39a48f-277c-477e-805b-4155d27e8b2b" (UID: "9b39a48f-277c-477e-805b-4155d27e8b2b"). InnerVolumeSpecName "kube-api-access-pzj4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:53.835248 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.835213 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8gbp\" (UniqueName: \"kubernetes.io/projected/eaa413bc-463a-47e4-97d6-d6ec315ef3aa-kube-api-access-n8gbp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:53.835248 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:53.835241 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzj4c\" (UniqueName: \"kubernetes.io/projected/9b39a48f-277c-477e-805b-4155d27e8b2b-kube-api-access-pzj4c\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:54.354809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.354777 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" Oct 30 00:58:54.354809 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.354796 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7" event={"ID":"9b39a48f-277c-477e-805b-4155d27e8b2b","Type":"ContainerDied","Data":"740a0f11d378571d7fc18039de26aa91a41fc22785acfb324dd01f9cb2c65c53"} Oct 30 00:58:54.355372 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.354853 4517 scope.go:122] "RemoveContainer" containerID="df9c55e8f75657fc394286f9934e4dd84eb8184641aa9cfb3af11481ff3094ea" Oct 30 00:58:54.356646 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.356620 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" event={"ID":"eaa413bc-463a-47e4-97d6-d6ec315ef3aa","Type":"ContainerDied","Data":"46140d00826b7711ab4534e051abc715da38f8ad198555d5eadb5ce17d5663a8"} Oct 30 00:58:54.356646 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.356638 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v" Oct 30 00:58:54.367223 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.367140 4517 scope.go:122] "RemoveContainer" containerID="8d0d82bb99648dc4ab1e47a4aef592867109b930ecd4449ec8041218a53e10f0" Oct 30 00:58:54.406352 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.406281 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7"] Oct 30 00:58:54.414094 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.414068 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-610065-66c456c5f5-rsst7"] Oct 30 00:58:54.441029 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.441001 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v"] Oct 30 00:58:54.444945 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:54.444925 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-610065-997f67946-54k2v"] Oct 30 00:58:55.211939 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:55.211901 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:56.237218 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.237186 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b39a48f-277c-477e-805b-4155d27e8b2b" path="/var/lib/kubelet/pods/9b39a48f-277c-477e-805b-4155d27e8b2b/volumes" Oct 30 00:58:56.237739 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.237695 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa413bc-463a-47e4-97d6-d6ec315ef3aa" path="/var/lib/kubelet/pods/eaa413bc-463a-47e4-97d6-d6ec315ef3aa/volumes" Oct 30 00:58:56.777501 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.777475 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:58:56.884701 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.884664 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-data\") pod \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " Oct 30 00:58:56.885066 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.885043 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-kube-api-access-n6vmf\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-kube-api-access-n6vmf\") pod \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " Oct 30 00:58:56.885194 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.885088 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-data" pod "dcae3bbb-d666-405f-8a3b-e7b2440711a8" (UID: "dcae3bbb-d666-405f-8a3b-e7b2440711a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:56.885194 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.885141 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-podinfo\") pod \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " Oct 30 00:58:56.885301 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.885291 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-envoy\") pod \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " Oct 30 00:58:56.886624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.885457 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-token\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-token\") pod \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " Oct 30 00:58:56.886624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.885496 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-socket\") pod \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " Oct 30 00:58:56.886624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.885547 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-certs\") pod \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " Oct 30 00:58:56.886624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.885586 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istiod-ca-cert\") pod \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " Oct 30 00:58:56.886624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.885625 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-credential-socket\") pod \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\" (UID: \"dcae3bbb-d666-405f-8a3b-e7b2440711a8\") " Oct 30 00:58:56.886624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.886448 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:56.886624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.886498 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-certs" pod "dcae3bbb-d666-405f-8a3b-e7b2440711a8" (UID: "dcae3bbb-d666-405f-8a3b-e7b2440711a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:56.886624 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.886560 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-socket" pod "dcae3bbb-d666-405f-8a3b-e7b2440711a8" (UID: "dcae3bbb-d666-405f-8a3b-e7b2440711a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:56.887350 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.886646 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-credential-socket" pod "dcae3bbb-d666-405f-8a3b-e7b2440711a8" (UID: "dcae3bbb-d666-405f-8a3b-e7b2440711a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:56.887350 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.886645 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istiod-ca-cert" pod "dcae3bbb-d666-405f-8a3b-e7b2440711a8" (UID: "dcae3bbb-d666-405f-8a3b-e7b2440711a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:56.889640 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.889573 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-podinfo" pod "dcae3bbb-d666-405f-8a3b-e7b2440711a8" (UID: "dcae3bbb-d666-405f-8a3b-e7b2440711a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:58:56.889640 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.889612 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-kube-api-access-n6vmf" pod "dcae3bbb-d666-405f-8a3b-e7b2440711a8" (UID: "dcae3bbb-d666-405f-8a3b-e7b2440711a8"). InnerVolumeSpecName "kube-api-access-n6vmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:56.890014 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.889689 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-token" pod "dcae3bbb-d666-405f-8a3b-e7b2440711a8" (UID: "dcae3bbb-d666-405f-8a3b-e7b2440711a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:56.891235 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.891211 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-envoy" pod "dcae3bbb-d666-405f-8a3b-e7b2440711a8" (UID: "dcae3bbb-d666-405f-8a3b-e7b2440711a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:56.987691 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.987651 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:56.987691 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.987690 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:56.987918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.987725 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:56.987918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.987739 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:56.987918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.987754 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:56.987918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.987767 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae3bbb-d666-405f-8a3b-e7b2440711a8-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:56.987918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.987780 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6vmf\" (UniqueName: \"kubernetes.io/projected/dcae3bbb-d666-405f-8a3b-e7b2440711a8-kube-api-access-n6vmf\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:56.987918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:56.987796 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae3bbb-d666-405f-8a3b-e7b2440711a8-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:58:57.220900 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.220856 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:57.399021 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.398984 4517 generic.go:356] "Generic (PLEG): container finished" podID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerID="de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32" exitCode=0 Oct 30 00:58:57.399523 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.399063 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" Oct 30 00:58:57.399523 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.399071 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" event={"ID":"dcae3bbb-d666-405f-8a3b-e7b2440711a8","Type":"ContainerDied","Data":"de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32"} Oct 30 00:58:57.399523 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.399145 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd" event={"ID":"dcae3bbb-d666-405f-8a3b-e7b2440711a8","Type":"ContainerDied","Data":"64983cc980e9f00523a4978803782ff52c7619bdbdb78b545b5e34e251cedcf5"} Oct 30 00:58:57.399523 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.399174 4517 scope.go:122] "RemoveContainer" containerID="de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32" Oct 30 00:58:57.411353 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.411337 4517 scope.go:122] "RemoveContainer" containerID="ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3" Oct 30 00:58:57.423018 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.422997 4517 scope.go:122] "RemoveContainer" containerID="de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32" Oct 30 00:58:57.423307 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:58:57.423288 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32\": container with ID starting with de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32 not found: ID does not exist" containerID="de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32" Oct 30 00:58:57.423364 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.423315 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32"} err="failed to get container status \"de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32\": rpc error: code = NotFound desc = could not find container \"de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32\": container with ID starting with de480bfebf8f480e04ceb00572c34c370b8681b8994a5625c7cdc391c0296a32 not found: ID does not exist" Oct 30 00:58:57.423364 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.423331 4517 scope.go:122] "RemoveContainer" containerID="ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3" Oct 30 00:58:57.423563 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:58:57.423546 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3\": container with ID starting with ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3 not found: ID does not exist" containerID="ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3" Oct 30 00:58:57.423609 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.423568 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3"} err="failed to get container status \"ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3\": rpc error: code = NotFound desc = could not find container \"ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3\": container with ID starting with ea63d5854037ed1d0b4a7a12bc8eb3fa59680f28127ccdaf7cb58fbfe5e413b3 not found: ID does not exist" Oct 30 00:58:57.460557 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.460533 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd"] Oct 30 00:58:57.470610 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:57.470586 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6867b8-openshift-default-54d69b94c9-9v5zd"] Oct 30 00:58:58.236821 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:58.236790 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" path="/var/lib/kubelet/pods/dcae3bbb-d666-405f-8a3b-e7b2440711a8/volumes" Oct 30 00:58:59.034152 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:58:59.034119 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:00.310243 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:00.310213 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:00.760678 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:59:00.760587 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:59:01.021658 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:01.021580 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:02.633918 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:02.633832 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:02.793971 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:02.793938 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:02.990450 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:02.990355 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:04.565997 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:04.565966 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:05.189160 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:05.189129 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:05.391413 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:05.391380 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:06.420106 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:06.420067 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:07.324672 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:07.324643 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:08.020925 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:08.020894 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:08.540886 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:08.540853 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:08.667213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:08.667185 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:10.151695 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:10.151659 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:11.824676 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:11.824645 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:13.413733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:13.413666 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:15.156869 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:15.156798 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:15.358039 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:15.358007 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:16.919542 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:16.919487 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:17.968463 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:17.968391 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:18.362100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:18.362066 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8"] Oct 30 00:59:18.362657 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:18.362624 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" podUID="d9daf3da-99a9-4ab1-9def-c2fde73335b0" containerName="maas-api" containerID="cri-o://8f2e75a1585002154987b4eb05579e48cd403a8db91f315e63e8eb80b8fe8bef" gracePeriod=30 Oct 30 00:59:18.430412 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:18.430325 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:19.076384 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.075881 4517 generic.go:356] "Generic (PLEG): container finished" podID="d9daf3da-99a9-4ab1-9def-c2fde73335b0" containerID="8f2e75a1585002154987b4eb05579e48cd403a8db91f315e63e8eb80b8fe8bef" exitCode=0 Oct 30 00:59:19.076384 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.076032 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" event={"ID":"d9daf3da-99a9-4ab1-9def-c2fde73335b0","Type":"ContainerDied","Data":"8f2e75a1585002154987b4eb05579e48cd403a8db91f315e63e8eb80b8fe8bef"} Oct 30 00:59:19.117787 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.109905 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:59:19.180761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.167786 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9daf3da-99a9-4ab1-9def-c2fde73335b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9daf3da-99a9-4ab1-9def-c2fde73335b0-openshift-service-ca\") pod \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " Oct 30 00:59:19.180761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.167850 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-maas-api-tls\") pod \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " Oct 30 00:59:19.180761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.168014 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9daf3da-99a9-4ab1-9def-c2fde73335b0-kube-api-access-5ztff\" (UniqueName: \"kubernetes.io/projected/d9daf3da-99a9-4ab1-9def-c2fde73335b0-kube-api-access-5ztff\") pod \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " Oct 30 00:59:19.180761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.168309 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-metrics-certs\") pod \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\" (UID: \"d9daf3da-99a9-4ab1-9def-c2fde73335b0\") " Oct 30 00:59:19.180761 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.170672 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9daf3da-99a9-4ab1-9def-c2fde73335b0-openshift-service-ca" pod "d9daf3da-99a9-4ab1-9def-c2fde73335b0" (UID: "d9daf3da-99a9-4ab1-9def-c2fde73335b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:19.196733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.190672 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-metrics-certs" pod "d9daf3da-99a9-4ab1-9def-c2fde73335b0" (UID: "d9daf3da-99a9-4ab1-9def-c2fde73335b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:19.196733 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.190861 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-maas-api-tls" pod "d9daf3da-99a9-4ab1-9def-c2fde73335b0" (UID: "d9daf3da-99a9-4ab1-9def-c2fde73335b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:19.200774 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.197904 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9daf3da-99a9-4ab1-9def-c2fde73335b0-kube-api-access-5ztff" pod "d9daf3da-99a9-4ab1-9def-c2fde73335b0" (UID: "d9daf3da-99a9-4ab1-9def-c2fde73335b0"). InnerVolumeSpecName "kube-api-access-5ztff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:19.272129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.272088 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9daf3da-99a9-4ab1-9def-c2fde73335b0-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:19.272129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.272143 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:19.272129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.272158 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ztff\" (UniqueName: \"kubernetes.io/projected/d9daf3da-99a9-4ab1-9def-c2fde73335b0-kube-api-access-5ztff\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:19.272465 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.272171 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9daf3da-99a9-4ab1-9def-c2fde73335b0-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:19.670414 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.670290 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:19.877686 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.877634 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:19.992605 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:19.992432 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:20.094488 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:20.093405 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" event={"ID":"d9daf3da-99a9-4ab1-9def-c2fde73335b0","Type":"ContainerDied","Data":"5aa8eb521f1fde4f9b23f7aa5ada57316fad015d3713ba14b1b2e3d413248a40"} Oct 30 00:59:20.094488 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:20.093476 4517 scope.go:122] "RemoveContainer" containerID="8f2e75a1585002154987b4eb05579e48cd403a8db91f315e63e8eb80b8fe8bef" Oct 30 00:59:20.094488 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:20.093633 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8" Oct 30 00:59:20.149515 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:20.149483 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8"] Oct 30 00:59:20.161051 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:20.161024 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a4fe0143-5b969575-4hdf8"] Oct 30 00:59:20.238926 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:20.238893 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9daf3da-99a9-4ab1-9def-c2fde73335b0" path="/var/lib/kubelet/pods/d9daf3da-99a9-4ab1-9def-c2fde73335b0/volumes" Oct 30 00:59:20.425303 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:20.425268 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:21.804207 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:21.804161 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:21.980627 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:21.980598 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:22.101071 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:22.100993 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:23.402129 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:23.402086 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:23.533549 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:23.533515 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:23.908881 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:23.899522 4517 scope.go:122] "RemoveContainer" containerID="97aa5ed0ed520b79d2ac4e420ca4e62274006f47b31d13d41c87fbb9da26798c" Oct 30 00:59:23.938047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:23.938022 4517 scope.go:122] "RemoveContainer" containerID="2f33f70e04ce59c192a1efb2dd135fc33ceeac0b86e6fb2c977401240bdd5ffa" Oct 30 00:59:24.248157 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:24.246607 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:25.964269 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:25.964220 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:27.033327 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:27.033286 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:27.152106 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:27.152070 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:27.382965 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:27.382928 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:28.067659 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:28.067604 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:28.216305 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:28.216272 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:29.160697 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:29.160625 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:29.400023 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:29.399984 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:30.399226 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:30.399197 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:30.760792 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:59:30.760687 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 00:59:32.234492 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:32.234448 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:32.772391 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:32.772352 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:32.928168 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:32.928111 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:34.312676 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:34.312641 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:34.469014 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:34.468952 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:34.625947 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:34.625857 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:36.247139 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:36.247102 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:37.900844 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:37.900803 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:39.156743 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:39.156691 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw"] Oct 30 00:59:39.189431 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:39.189395 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw"] Oct 30 00:59:39.303684 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:39.303652 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:39.422094 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:39.421995 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:39.833280 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:39.833245 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:41.646204 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:41.645840 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:43.186919 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:43.186888 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j"] Oct 30 00:59:43.188378 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:43.188334 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" podUID="cba5971c-13b9-419e-af84-2d49cd38b2b5" containerName="maas-api" containerID="cri-o://17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37" gracePeriod=30 Oct 30 00:59:43.245753 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:43.245723 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:43.963746 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:43.950233 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:43.981031 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:43.980998 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:59:44.088988 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.088955 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:44.110672 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.110649 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-maas-api-tls\") pod \"cba5971c-13b9-419e-af84-2d49cd38b2b5\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " Oct 30 00:59:44.110839 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.110766 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba5971c-13b9-419e-af84-2d49cd38b2b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba5971c-13b9-419e-af84-2d49cd38b2b5-openshift-service-ca\") pod \"cba5971c-13b9-419e-af84-2d49cd38b2b5\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " Oct 30 00:59:44.110902 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.110839 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba5971c-13b9-419e-af84-2d49cd38b2b5-kube-api-access-4t889\" (UniqueName: \"kubernetes.io/projected/cba5971c-13b9-419e-af84-2d49cd38b2b5-kube-api-access-4t889\") pod \"cba5971c-13b9-419e-af84-2d49cd38b2b5\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " Oct 30 00:59:44.111107 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.111086 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-metrics-certs\") pod \"cba5971c-13b9-419e-af84-2d49cd38b2b5\" (UID: \"cba5971c-13b9-419e-af84-2d49cd38b2b5\") " Oct 30 00:59:44.111238 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.111160 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba5971c-13b9-419e-af84-2d49cd38b2b5-openshift-service-ca" pod "cba5971c-13b9-419e-af84-2d49cd38b2b5" (UID: "cba5971c-13b9-419e-af84-2d49cd38b2b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:44.111957 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.111877 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba5971c-13b9-419e-af84-2d49cd38b2b5-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:44.114123 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.114101 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-maas-api-tls" pod "cba5971c-13b9-419e-af84-2d49cd38b2b5" (UID: "cba5971c-13b9-419e-af84-2d49cd38b2b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:44.114310 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.114171 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba5971c-13b9-419e-af84-2d49cd38b2b5-kube-api-access-4t889" pod "cba5971c-13b9-419e-af84-2d49cd38b2b5" (UID: "cba5971c-13b9-419e-af84-2d49cd38b2b5"). InnerVolumeSpecName "kube-api-access-4t889". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:44.114440 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.114186 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-metrics-certs" pod "cba5971c-13b9-419e-af84-2d49cd38b2b5" (UID: "cba5971c-13b9-419e-af84-2d49cd38b2b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:44.157641 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.157527 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" podUID="3f998c5a-b2c5-4f94-81ff-875400c91125" containerName="payload-processing" containerID="cri-o://f5d0a36ef67e29743aacde6b722af0bde317675bcbc9720d848ea262c1cf61e0" gracePeriod=25 Oct 30 00:59:44.190369 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.190307 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" podUID="1cd797a2-2e92-4084-b06a-1318ebd694cb" containerName="payload-pre-processing" containerID="cri-o://c8c78df6c4c2d9b0230c288cbde03446ca5f5f8b31a406c889e28a913509343e" gracePeriod=25 Oct 30 00:59:44.213294 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.213260 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t889\" (UniqueName: \"kubernetes.io/projected/cba5971c-13b9-419e-af84-2d49cd38b2b5-kube-api-access-4t889\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:44.213294 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.213287 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:44.213294 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.213298 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cba5971c-13b9-419e-af84-2d49cd38b2b5-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:44.289364 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.289327 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:44.313779 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.313689 4517 generic.go:356] "Generic (PLEG): container finished" podID="3f998c5a-b2c5-4f94-81ff-875400c91125" containerID="f5d0a36ef67e29743aacde6b722af0bde317675bcbc9720d848ea262c1cf61e0" exitCode=0 Oct 30 00:59:44.313998 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.313896 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" event={"ID":"3f998c5a-b2c5-4f94-81ff-875400c91125","Type":"ContainerDied","Data":"f5d0a36ef67e29743aacde6b722af0bde317675bcbc9720d848ea262c1cf61e0"} Oct 30 00:59:44.316188 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.316158 4517 generic.go:356] "Generic (PLEG): container finished" podID="1cd797a2-2e92-4084-b06a-1318ebd694cb" containerID="c8c78df6c4c2d9b0230c288cbde03446ca5f5f8b31a406c889e28a913509343e" exitCode=0 Oct 30 00:59:44.316295 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.316235 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" event={"ID":"1cd797a2-2e92-4084-b06a-1318ebd694cb","Type":"ContainerDied","Data":"c8c78df6c4c2d9b0230c288cbde03446ca5f5f8b31a406c889e28a913509343e"} Oct 30 00:59:44.317887 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.317863 4517 generic.go:356] "Generic (PLEG): container finished" podID="cba5971c-13b9-419e-af84-2d49cd38b2b5" containerID="17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37" exitCode=0 Oct 30 00:59:44.318022 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.317920 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" event={"ID":"cba5971c-13b9-419e-af84-2d49cd38b2b5","Type":"ContainerDied","Data":"17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37"} Oct 30 00:59:44.318022 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.317928 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" Oct 30 00:59:44.318022 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.317958 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j" event={"ID":"cba5971c-13b9-419e-af84-2d49cd38b2b5","Type":"ContainerDied","Data":"76f89cdc32df22a014f23f391f16142f6e7ea9f4ee0b7d11d390ef4e4bedcb49"} Oct 30 00:59:44.318022 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.317984 4517 scope.go:122] "RemoveContainer" containerID="17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37" Oct 30 00:59:44.339065 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.338782 4517 scope.go:122] "RemoveContainer" containerID="17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37" Oct 30 00:59:44.339651 ip-10-0-132-84 kubenswrapper[4517]: E1030 00:59:44.339566 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37\": container with ID starting with 17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37 not found: ID does not exist" containerID="17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37" Oct 30 00:59:44.339651 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.339608 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37"} err="failed to get container status \"17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37\": rpc error: code = NotFound desc = could not find container \"17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37\": container with ID starting with 17136b57ae735ab740385803b6b391e3cd526dd1c67cd389c4532ecae16b0d37 not found: ID does not exist" Oct 30 00:59:44.386677 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.378383 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j"] Oct 30 00:59:44.386677 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.385626 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-610065-cd54c8ff6-fkt5j"] Oct 30 00:59:44.584742 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.584521 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:59:44.637655 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.637628 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:59:44.651077 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.651048 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:44.728893 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.728861 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f998c5a-b2c5-4f94-81ff-875400c91125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f998c5a-b2c5-4f94-81ff-875400c91125-plugins-config-volume\") pod \"3f998c5a-b2c5-4f94-81ff-875400c91125\" (UID: \"3f998c5a-b2c5-4f94-81ff-875400c91125\") " Oct 30 00:59:44.729179 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.729161 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd797a2-2e92-4084-b06a-1318ebd694cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd797a2-2e92-4084-b06a-1318ebd694cb-plugins-config-volume\") pod \"1cd797a2-2e92-4084-b06a-1318ebd694cb\" (UID: \"1cd797a2-2e92-4084-b06a-1318ebd694cb\") " Oct 30 00:59:44.729276 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.729219 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd797a2-2e92-4084-b06a-1318ebd694cb-kube-api-access-b6dms\" (UniqueName: \"kubernetes.io/projected/1cd797a2-2e92-4084-b06a-1318ebd694cb-kube-api-access-b6dms\") pod \"1cd797a2-2e92-4084-b06a-1318ebd694cb\" (UID: \"1cd797a2-2e92-4084-b06a-1318ebd694cb\") " Oct 30 00:59:44.729276 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.729241 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f998c5a-b2c5-4f94-81ff-875400c91125-plugins-config-volume" pod "3f998c5a-b2c5-4f94-81ff-875400c91125" (UID: "3f998c5a-b2c5-4f94-81ff-875400c91125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:44.729460 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.729342 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f998c5a-b2c5-4f94-81ff-875400c91125-kube-api-access-f7zkl\" (UniqueName: \"kubernetes.io/projected/3f998c5a-b2c5-4f94-81ff-875400c91125-kube-api-access-f7zkl\") pod \"3f998c5a-b2c5-4f94-81ff-875400c91125\" (UID: \"3f998c5a-b2c5-4f94-81ff-875400c91125\") " Oct 30 00:59:44.729928 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.729675 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd797a2-2e92-4084-b06a-1318ebd694cb-plugins-config-volume" pod "1cd797a2-2e92-4084-b06a-1318ebd694cb" (UID: "1cd797a2-2e92-4084-b06a-1318ebd694cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:44.730043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.729988 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd797a2-2e92-4084-b06a-1318ebd694cb-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:44.730043 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.730008 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f998c5a-b2c5-4f94-81ff-875400c91125-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:44.731942 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.731918 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f998c5a-b2c5-4f94-81ff-875400c91125-kube-api-access-f7zkl" pod "3f998c5a-b2c5-4f94-81ff-875400c91125" (UID: "3f998c5a-b2c5-4f94-81ff-875400c91125"). InnerVolumeSpecName "kube-api-access-f7zkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:44.732213 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.732191 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd797a2-2e92-4084-b06a-1318ebd694cb-kube-api-access-b6dms" pod "1cd797a2-2e92-4084-b06a-1318ebd694cb" (UID: "1cd797a2-2e92-4084-b06a-1318ebd694cb"). InnerVolumeSpecName "kube-api-access-b6dms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:44.831169 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.831138 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6dms\" (UniqueName: \"kubernetes.io/projected/1cd797a2-2e92-4084-b06a-1318ebd694cb-kube-api-access-b6dms\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:44.831169 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:44.831164 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7zkl\" (UniqueName: \"kubernetes.io/projected/3f998c5a-b2c5-4f94-81ff-875400c91125-kube-api-access-f7zkl\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:45.328651 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.328622 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" Oct 30 00:59:45.329220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.328655 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw" event={"ID":"3f998c5a-b2c5-4f94-81ff-875400c91125","Type":"ContainerDied","Data":"fe04a0bb6fb02a140d92285cad99f2a44ecfa41041d40567d3e8fabb652b94be"} Oct 30 00:59:45.329220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.328732 4517 scope.go:122] "RemoveContainer" containerID="f5d0a36ef67e29743aacde6b722af0bde317675bcbc9720d848ea262c1cf61e0" Oct 30 00:59:45.330528 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.330510 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" Oct 30 00:59:45.330649 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.330535 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw" event={"ID":"1cd797a2-2e92-4084-b06a-1318ebd694cb","Type":"ContainerDied","Data":"0230584cebf7743cf7e03d9e6d4c90071a2401eb9294ae3b8b24a9c3cf992e07"} Oct 30 00:59:45.340794 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.340774 4517 scope.go:122] "RemoveContainer" containerID="c8c78df6c4c2d9b0230c288cbde03446ca5f5f8b31a406c889e28a913509343e" Oct 30 00:59:45.393937 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.393912 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw"] Oct 30 00:59:45.404510 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.404488 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8090513-69dcffc55b-xgmzw"] Oct 30 00:59:45.434496 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.434454 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw"] Oct 30 00:59:45.451461 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:45.451434 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-5xchw"] Oct 30 00:59:46.224458 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:46.224427 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:46.235717 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:46.235686 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd797a2-2e92-4084-b06a-1318ebd694cb" path="/var/lib/kubelet/pods/1cd797a2-2e92-4084-b06a-1318ebd694cb/volumes" Oct 30 00:59:46.236158 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:46.236145 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f998c5a-b2c5-4f94-81ff-875400c91125" path="/var/lib/kubelet/pods/3f998c5a-b2c5-4f94-81ff-875400c91125/volumes" Oct 30 00:59:46.236539 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:46.236527 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba5971c-13b9-419e-af84-2d49cd38b2b5" path="/var/lib/kubelet/pods/cba5971c-13b9-419e-af84-2d49cd38b2b5/volumes" Oct 30 00:59:47.881220 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:47.881192 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:49.521354 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:49.521317 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:50.824831 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:50.824794 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq"] Oct 30 00:59:50.825349 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:50.825019 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" podUID="72eec295-5b8b-42e8-9484-da7c6e892669" containerName="maas-api" containerID="cri-o://1b8f3a0a9cd509c140cd69109c7a3dd3af00a5e8d97985a54f6e187535149e3c" gracePeriod=30 Oct 30 00:59:50.920512 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:50.920479 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" podUID="72eec295-5b8b-42e8-9484-da7c6e892669" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.102:8443/health\": dial tcp 10.133.0.102:8443: connect: connection refused" Oct 30 00:59:51.106801 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:51.055192 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:51.276750 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:51.271724 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:51.486254 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:51.484628 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:51.492387 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:51.492358 4517 generic.go:356] "Generic (PLEG): container finished" podID="72eec295-5b8b-42e8-9484-da7c6e892669" containerID="1b8f3a0a9cd509c140cd69109c7a3dd3af00a5e8d97985a54f6e187535149e3c" exitCode=0 Oct 30 00:59:51.492725 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:51.492682 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" event={"ID":"72eec295-5b8b-42e8-9484-da7c6e892669","Type":"ContainerDied","Data":"1b8f3a0a9cd509c140cd69109c7a3dd3af00a5e8d97985a54f6e187535149e3c"} Oct 30 00:59:51.621583 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:51.621548 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:51.771377 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:51.771335 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:51.896730 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:51.884423 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:59:52.064728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.059278 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-maas-api-tls\") pod \"72eec295-5b8b-42e8-9484-da7c6e892669\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " Oct 30 00:59:52.064728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.059366 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-metrics-certs\") pod \"72eec295-5b8b-42e8-9484-da7c6e892669\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " Oct 30 00:59:52.064728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.059454 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72eec295-5b8b-42e8-9484-da7c6e892669-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72eec295-5b8b-42e8-9484-da7c6e892669-openshift-service-ca\") pod \"72eec295-5b8b-42e8-9484-da7c6e892669\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " Oct 30 00:59:52.064728 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.059644 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72eec295-5b8b-42e8-9484-da7c6e892669-kube-api-access-4ghcf\" (UniqueName: \"kubernetes.io/projected/72eec295-5b8b-42e8-9484-da7c6e892669-kube-api-access-4ghcf\") pod \"72eec295-5b8b-42e8-9484-da7c6e892669\" (UID: \"72eec295-5b8b-42e8-9484-da7c6e892669\") " Oct 30 00:59:52.098079 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.095779 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-maas-api-tls" pod "72eec295-5b8b-42e8-9484-da7c6e892669" (UID: "72eec295-5b8b-42e8-9484-da7c6e892669"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:52.099985 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.098398 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72eec295-5b8b-42e8-9484-da7c6e892669-openshift-service-ca" pod "72eec295-5b8b-42e8-9484-da7c6e892669" (UID: "72eec295-5b8b-42e8-9484-da7c6e892669"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:52.102964 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.102937 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72eec295-5b8b-42e8-9484-da7c6e892669-kube-api-access-4ghcf" pod "72eec295-5b8b-42e8-9484-da7c6e892669" (UID: "72eec295-5b8b-42e8-9484-da7c6e892669"). InnerVolumeSpecName "kube-api-access-4ghcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:52.114939 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.114202 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-metrics-certs" pod "72eec295-5b8b-42e8-9484-da7c6e892669" (UID: "72eec295-5b8b-42e8-9484-da7c6e892669"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:52.164680 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.164596 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ghcf\" (UniqueName: \"kubernetes.io/projected/72eec295-5b8b-42e8-9484-da7c6e892669-kube-api-access-4ghcf\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:52.164996 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.164975 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:52.165106 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.165094 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72eec295-5b8b-42e8-9484-da7c6e892669-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:52.165200 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.165189 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72eec295-5b8b-42e8-9484-da7c6e892669-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 00:59:52.325137 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.325037 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:52.501786 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.501753 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" Oct 30 00:59:52.501786 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.501766 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq" event={"ID":"72eec295-5b8b-42e8-9484-da7c6e892669","Type":"ContainerDied","Data":"ee21f2c4bd41eea8ce04ab417f0f025a5d46d8ba9b5aa4ebec613232064c2a03"} Oct 30 00:59:52.502047 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.501838 4517 scope.go:122] "RemoveContainer" containerID="1b8f3a0a9cd509c140cd69109c7a3dd3af00a5e8d97985a54f6e187535149e3c" Oct 30 00:59:52.548792 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.548768 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq"] Oct 30 00:59:52.558961 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.558939 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-353403-6f9bcf884b-78txq"] Oct 30 00:59:52.816896 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.816865 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:52.926568 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:52.926533 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:53.043155 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:53.043121 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:53.465100 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:53.465068 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:54.236106 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:54.236069 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72eec295-5b8b-42e8-9484-da7c6e892669" path="/var/lib/kubelet/pods/72eec295-5b8b-42e8-9484-da7c6e892669/volumes" Oct 30 00:59:54.985241 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:54.985212 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:55.105847 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:55.105806 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:57.068028 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:57.067992 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:58.424250 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:58.424219 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt"] Oct 30 00:59:58.514381 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:58.514346 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:58.729602 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:58.729520 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:59.023357 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:59.023269 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:59.156626 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:59.156593 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:59.317488 ip-10-0-132-84 kubenswrapper[4517]: I1030 00:59:59.317455 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:00.165035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165000 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc"] Oct 30 01:00:00.165666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165647 4517 cpu_manager.go:414] "containerMap: removing container" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" Oct 30 01:00:00.165666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165666 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165677 4517 cpu_manager.go:414] "containerMap: removing container" podUID="cba5971c-13b9-419e-af84-2d49cd38b2b5" containerName="maas-api" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165683 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba5971c-13b9-419e-af84-2d49cd38b2b5" containerName="maas-api" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165700 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1cd797a2-2e92-4084-b06a-1318ebd694cb" containerName="payload-pre-processing" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165721 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd797a2-2e92-4084-b06a-1318ebd694cb" containerName="payload-pre-processing" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165746 4517 cpu_manager.go:414] "containerMap: removing container" podUID="9b39a48f-277c-477e-805b-4155d27e8b2b" containerName="payload-pre-processing" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165753 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b39a48f-277c-477e-805b-4155d27e8b2b" containerName="payload-pre-processing" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165773 4517 cpu_manager.go:414] "containerMap: removing container" podUID="72eec295-5b8b-42e8-9484-da7c6e892669" containerName="maas-api" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165778 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72eec295-5b8b-42e8-9484-da7c6e892669" containerName="maas-api" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165797 4517 cpu_manager.go:414] "containerMap: removing container" podUID="71412d84-5e97-4794-aea9-4418023285d0" containerName="maas-api" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165801 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71412d84-5e97-4794-aea9-4418023285d0" containerName="maas-api" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165807 4517 cpu_manager.go:414] "containerMap: removing container" podUID="eaa413bc-463a-47e4-97d6-d6ec315ef3aa" containerName="payload-processing" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165811 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa413bc-463a-47e4-97d6-d6ec315ef3aa" containerName="payload-processing" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165823 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d9daf3da-99a9-4ab1-9def-c2fde73335b0" containerName="maas-api" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165827 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9daf3da-99a9-4ab1-9def-c2fde73335b0" containerName="maas-api" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165833 4517 cpu_manager.go:414] "containerMap: removing container" podUID="3f998c5a-b2c5-4f94-81ff-875400c91125" containerName="payload-processing" Oct 30 01:00:00.165841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165837 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f998c5a-b2c5-4f94-81ff-875400c91125" containerName="payload-processing" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165954 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b39a48f-277c-477e-805b-4155d27e8b2b" containerName="payload-pre-processing" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165967 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="72eec295-5b8b-42e8-9484-da7c6e892669" containerName="maas-api" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165975 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="71412d84-5e97-4794-aea9-4418023285d0" containerName="maas-api" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165984 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa413bc-463a-47e4-97d6-d6ec315ef3aa" containerName="payload-processing" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.165994 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9daf3da-99a9-4ab1-9def-c2fde73335b0" containerName="maas-api" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.166019 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.166027 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f998c5a-b2c5-4f94-81ff-875400c91125" containerName="payload-processing" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.166038 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.166047 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba5971c-13b9-419e-af84-2d49cd38b2b5" containerName="maas-api" Oct 30 01:00:00.166623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.166059 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd797a2-2e92-4084-b06a-1318ebd694cb" containerName="payload-pre-processing" Oct 30 01:00:00.171185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.171160 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v"] Oct 30 01:00:00.171348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.171298 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:00.171957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.171938 4517 cpu_manager.go:414] "containerMap: removing container" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" Oct 30 01:00:00.171957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.171955 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcae3bbb-d666-405f-8a3b-e7b2440711a8" containerName="istio-proxy" Oct 30 01:00:00.175216 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.175197 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:00.187250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.187225 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm"] Oct 30 01:00:00.194052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.194030 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:00.198295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.198271 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk"] Oct 30 01:00:00.205437 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.205414 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:00.208839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.208814 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:00:00.208979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.208858 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz"] Oct 30 01:00:00.211727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.211681 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc\" (UID: \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:00.211913 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.211887 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvglr\" (UniqueName: \"kubernetes.io/projected/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-kube-api-access-fvglr\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc\" (UID: \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:00.212039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.212008 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3e6997-7ad8-4065-96ea-78928efa2227-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v\" (UID: \"fd3e6997-7ad8-4065-96ea-78928efa2227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:00.212173 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.212055 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nrjl\" (UniqueName: \"kubernetes.io/projected/5861cf21-9842-4a34-884d-85ce63b421ee-kube-api-access-4nrjl\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk\" (UID: \"5861cf21-9842-4a34-884d-85ce63b421ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:00.212173 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.212128 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1687a186-3e99-47e3-8b70-d131495afade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm\" (UID: \"1687a186-3e99-47e3-8b70-d131495afade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:00.212173 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.212151 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtbd\" (UniqueName: \"kubernetes.io/projected/1687a186-3e99-47e3-8b70-d131495afade-kube-api-access-rgtbd\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm\" (UID: \"1687a186-3e99-47e3-8b70-d131495afade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:00.212409 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.212184 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw4l7\" (UniqueName: \"kubernetes.io/projected/fd3e6997-7ad8-4065-96ea-78928efa2227-kube-api-access-jw4l7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v\" (UID: \"fd3e6997-7ad8-4065-96ea-78928efa2227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:00.212409 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.212228 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5861cf21-9842-4a34-884d-85ce63b421ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk\" (UID: \"5861cf21-9842-4a34-884d-85ce63b421ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:00.216731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.216695 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc"] Oct 30 01:00:00.216974 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.216959 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:00.247979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.246381 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v"] Oct 30 01:00:00.250688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.250534 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm"] Oct 30 01:00:00.276524 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.271536 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk"] Oct 30 01:00:00.276524 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.271612 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz"] Oct 30 01:00:00.280814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.279309 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z"] Oct 30 01:00:00.287511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.284316 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm"] Oct 30 01:00:00.287511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.284552 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:00.296720 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.296520 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:00.305778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.300549 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt"] Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.315371 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9nl\" (UniqueName: \"kubernetes.io/projected/c15581be-8d44-43be-832f-323c013ba2c2-kube-api-access-7x9nl\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz\" (UID: \"c15581be-8d44-43be-832f-323c013ba2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.315432 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3e6997-7ad8-4065-96ea-78928efa2227-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v\" (UID: \"fd3e6997-7ad8-4065-96ea-78928efa2227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.315540 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nrjl\" (UniqueName: \"kubernetes.io/projected/5861cf21-9842-4a34-884d-85ce63b421ee-kube-api-access-4nrjl\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk\" (UID: \"5861cf21-9842-4a34-884d-85ce63b421ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.315595 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm\" (UID: \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.315658 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1687a186-3e99-47e3-8b70-d131495afade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm\" (UID: \"1687a186-3e99-47e3-8b70-d131495afade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.315682 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtbd\" (UniqueName: \"kubernetes.io/projected/1687a186-3e99-47e3-8b70-d131495afade-kube-api-access-rgtbd\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm\" (UID: \"1687a186-3e99-47e3-8b70-d131495afade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.315806 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw4l7\" (UniqueName: \"kubernetes.io/projected/fd3e6997-7ad8-4065-96ea-78928efa2227-kube-api-access-jw4l7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v\" (UID: \"fd3e6997-7ad8-4065-96ea-78928efa2227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.315883 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5861cf21-9842-4a34-884d-85ce63b421ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk\" (UID: \"5861cf21-9842-4a34-884d-85ce63b421ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.316133 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dblj\" (UniqueName: \"kubernetes.io/projected/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-kube-api-access-7dblj\") pod \"maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z\" (UID: \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.316186 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15581be-8d44-43be-832f-323c013ba2c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz\" (UID: \"c15581be-8d44-43be-832f-323c013ba2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.316221 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc\" (UID: \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.316326 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9vb\" (UniqueName: \"kubernetes.io/projected/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-kube-api-access-xk9vb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm\" (UID: \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.316554 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvglr\" (UniqueName: \"kubernetes.io/projected/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-kube-api-access-fvglr\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc\" (UID: \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:00.317250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.316603 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z\" (UID: \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:00.318915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.318499 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1687a186-3e99-47e3-8b70-d131495afade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm\" (UID: \"1687a186-3e99-47e3-8b70-d131495afade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:00.318915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.318679 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc\" (UID: \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:00.320762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.320018 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3e6997-7ad8-4065-96ea-78928efa2227-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v\" (UID: \"fd3e6997-7ad8-4065-96ea-78928efa2227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:00.323036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.323009 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5861cf21-9842-4a34-884d-85ce63b421ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk\" (UID: \"5861cf21-9842-4a34-884d-85ce63b421ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:00.323275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.323256 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z"] Oct 30 01:00:00.323368 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.323311 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm"] Oct 30 01:00:00.323565 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.323549 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:00.330611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.330595 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt"] Oct 30 01:00:00.335539 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.335519 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t"] Oct 30 01:00:00.345214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.345194 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t"] Oct 30 01:00:00.345329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.345264 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:00.353603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.353574 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtbd\" (UniqueName: \"kubernetes.io/projected/1687a186-3e99-47e3-8b70-d131495afade-kube-api-access-rgtbd\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm\" (UID: \"1687a186-3e99-47e3-8b70-d131495afade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:00.354322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.354296 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw4l7\" (UniqueName: \"kubernetes.io/projected/fd3e6997-7ad8-4065-96ea-78928efa2227-kube-api-access-jw4l7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v\" (UID: \"fd3e6997-7ad8-4065-96ea-78928efa2227\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:00.355428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.355400 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvglr\" (UniqueName: \"kubernetes.io/projected/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-kube-api-access-fvglr\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc\" (UID: \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:00.357399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.357368 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nrjl\" (UniqueName: \"kubernetes.io/projected/5861cf21-9842-4a34-884d-85ce63b421ee-kube-api-access-4nrjl\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk\" (UID: \"5861cf21-9842-4a34-884d-85ce63b421ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:00.419079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419012 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z\" (UID: \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:00.419213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419095 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696d68c-2f3d-4a6a-af56-7228baf535af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt\" (UID: \"5696d68c-2f3d-4a6a-af56-7228baf535af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:00.419213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419127 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9nl\" (UniqueName: \"kubernetes.io/projected/c15581be-8d44-43be-832f-323c013ba2c2-kube-api-access-7x9nl\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz\" (UID: \"c15581be-8d44-43be-832f-323c013ba2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:00.419213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419192 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm\" (UID: \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:00.419367 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419337 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dblj\" (UniqueName: \"kubernetes.io/projected/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-kube-api-access-7dblj\") pod \"maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z\" (UID: \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:00.419424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419388 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15581be-8d44-43be-832f-323c013ba2c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz\" (UID: \"c15581be-8d44-43be-832f-323c013ba2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:00.419474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419442 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9vb\" (UniqueName: \"kubernetes.io/projected/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-kube-api-access-xk9vb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm\" (UID: \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:00.419527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419502 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbcpp\" (UniqueName: \"kubernetes.io/projected/113b92e9-3769-4c4d-9a1d-e4f81277c68f-kube-api-access-xbcpp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t\" (UID: \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:00.419614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419595 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzj2h\" (UniqueName: \"kubernetes.io/projected/5696d68c-2f3d-4a6a-af56-7228baf535af-kube-api-access-rzj2h\") pod \"maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt\" (UID: \"5696d68c-2f3d-4a6a-af56-7228baf535af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:00.419740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419637 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113b92e9-3769-4c4d-9a1d-e4f81277c68f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t\" (UID: \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:00.419868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419845 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z\" (UID: \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:00.419929 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.419878 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm\" (UID: \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:00.420236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.420184 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15581be-8d44-43be-832f-323c013ba2c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz\" (UID: \"c15581be-8d44-43be-832f-323c013ba2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:00.444446 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.444423 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dblj\" (UniqueName: \"kubernetes.io/projected/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-kube-api-access-7dblj\") pod \"maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z\" (UID: \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:00.444562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.444515 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9vb\" (UniqueName: \"kubernetes.io/projected/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-kube-api-access-xk9vb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm\" (UID: \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:00.444810 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.444781 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9nl\" (UniqueName: \"kubernetes.io/projected/c15581be-8d44-43be-832f-323c013ba2c2-kube-api-access-7x9nl\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz\" (UID: \"c15581be-8d44-43be-832f-323c013ba2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:00.495226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.495200 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:00.521471 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.521444 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696d68c-2f3d-4a6a-af56-7228baf535af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt\" (UID: \"5696d68c-2f3d-4a6a-af56-7228baf535af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:00.521737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.521698 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbcpp\" (UniqueName: \"kubernetes.io/projected/113b92e9-3769-4c4d-9a1d-e4f81277c68f-kube-api-access-xbcpp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t\" (UID: \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:00.521824 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.521808 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzj2h\" (UniqueName: \"kubernetes.io/projected/5696d68c-2f3d-4a6a-af56-7228baf535af-kube-api-access-rzj2h\") pod \"maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt\" (UID: \"5696d68c-2f3d-4a6a-af56-7228baf535af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:00.521894 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.521855 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113b92e9-3769-4c4d-9a1d-e4f81277c68f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t\" (UID: \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:00.522095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.522072 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696d68c-2f3d-4a6a-af56-7228baf535af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt\" (UID: \"5696d68c-2f3d-4a6a-af56-7228baf535af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:00.522441 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.522407 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113b92e9-3769-4c4d-9a1d-e4f81277c68f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t\" (UID: \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:00.532567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.532545 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:00.536388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.536366 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:00.547121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.547095 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzj2h\" (UniqueName: \"kubernetes.io/projected/5696d68c-2f3d-4a6a-af56-7228baf535af-kube-api-access-rzj2h\") pod \"maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt\" (UID: \"5696d68c-2f3d-4a6a-af56-7228baf535af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:00.548488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.548444 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbcpp\" (UniqueName: \"kubernetes.io/projected/113b92e9-3769-4c4d-9a1d-e4f81277c68f-kube-api-access-xbcpp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t\" (UID: \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:00.579440 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.576968 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:00.607068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.607040 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:00.641545 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.641409 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:00.645810 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.645598 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:00.651961 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.651786 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:00.700217 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.699981 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:00.748358 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.748288 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:00.848792 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:00.847509 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:00:00.911942 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:00.911901 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc"] Oct 30 01:00:01.090354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.090324 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v"] Oct 30 01:00:01.106166 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.106141 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm"] Oct 30 01:00:01.135640 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.135615 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk"] Oct 30 01:00:01.169350 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:00:01.168667 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3e6997_7ad8_4065_96ea_78928efa2227.slice/crio-1b0e74ab5346ce76745ae9e80d33261bd00ad0d32cbc3a614576d6fefee7a90c WatchSource:0}: Error finding container 1b0e74ab5346ce76745ae9e80d33261bd00ad0d32cbc3a614576d6fefee7a90c: Status 404 returned error can't find the container with id 1b0e74ab5346ce76745ae9e80d33261bd00ad0d32cbc3a614576d6fefee7a90c Oct 30 01:00:01.202869 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.202249 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz"] Oct 30 01:00:01.210054 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:00:01.210025 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15581be_8d44_43be_832f_323c013ba2c2.slice/crio-531b05b3020fd9988bc099df7e2a00065c9bd0d0a5f2f8890d14817df987e9e5 WatchSource:0}: Error finding container 531b05b3020fd9988bc099df7e2a00065c9bd0d0a5f2f8890d14817df987e9e5: Status 404 returned error can't find the container with id 531b05b3020fd9988bc099df7e2a00065c9bd0d0a5f2f8890d14817df987e9e5 Oct 30 01:00:01.314338 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.314307 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z"] Oct 30 01:00:01.315754 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:00:01.315728 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb7aed0_c8eb_4969_a2c6_b1ed09b4e1a5.slice/crio-e255e922fdf3aec255c4028546ea7a42e02596299f4fb40e1a3eb4ca53cde344 WatchSource:0}: Error finding container e255e922fdf3aec255c4028546ea7a42e02596299f4fb40e1a3eb4ca53cde344: Status 404 returned error can't find the container with id e255e922fdf3aec255c4028546ea7a42e02596299f4fb40e1a3eb4ca53cde344 Oct 30 01:00:01.340918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.340895 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm"] Oct 30 01:00:01.347959 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:00:01.347937 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7ba61a_e155_4f51_9e2d_8ba7ca6c3972.slice/crio-a5fba76573b912a53e8f25eb8f37a1ee4f4a52a3b68e83b742adde036a3e07a0 WatchSource:0}: Error finding container a5fba76573b912a53e8f25eb8f37a1ee4f4a52a3b68e83b742adde036a3e07a0: Status 404 returned error can't find the container with id a5fba76573b912a53e8f25eb8f37a1ee4f4a52a3b68e83b742adde036a3e07a0 Oct 30 01:00:01.387996 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.387969 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt"] Oct 30 01:00:01.393043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.393021 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t"] Oct 30 01:00:01.467194 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:00:01.467115 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5696d68c_2f3d_4a6a_af56_7228baf535af.slice/crio-27c342ac9597f924951ed59a3170a58db10cad21d919a5aae43469e9012ee0fa WatchSource:0}: Error finding container 27c342ac9597f924951ed59a3170a58db10cad21d919a5aae43469e9012ee0fa: Status 404 returned error can't find the container with id 27c342ac9597f924951ed59a3170a58db10cad21d919a5aae43469e9012ee0fa Oct 30 01:00:01.467602 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:00:01.467567 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113b92e9_3769_4c4d_9a1d_e4f81277c68f.slice/crio-b5b92400b462389be58a87713081654fcdabc4b6d0fd9ea5b95ed68e90749e1a WatchSource:0}: Error finding container b5b92400b462389be58a87713081654fcdabc4b6d0fd9ea5b95ed68e90749e1a: Status 404 returned error can't find the container with id b5b92400b462389be58a87713081654fcdabc4b6d0fd9ea5b95ed68e90749e1a Oct 30 01:00:01.600526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.600497 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" event={"ID":"113b92e9-3769-4c4d-9a1d-e4f81277c68f","Type":"ContainerStarted","Data":"b5b92400b462389be58a87713081654fcdabc4b6d0fd9ea5b95ed68e90749e1a"} Oct 30 01:00:01.601979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.601953 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" event={"ID":"5696d68c-2f3d-4a6a-af56-7228baf535af","Type":"ContainerStarted","Data":"27c342ac9597f924951ed59a3170a58db10cad21d919a5aae43469e9012ee0fa"} Oct 30 01:00:01.603431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.603378 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" event={"ID":"1687a186-3e99-47e3-8b70-d131495afade","Type":"ContainerStarted","Data":"f691a735e5cbd98bd65a3f59a540952798e5ef4a72ecd55cf799a0ee344786cb"} Oct 30 01:00:01.604744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.604689 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" event={"ID":"5861cf21-9842-4a34-884d-85ce63b421ee","Type":"ContainerStarted","Data":"a02b326faaef94fff12ddc096560969bcdffc6befa3f7c0cd05d80acb93d9724"} Oct 30 01:00:01.606482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.606455 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" event={"ID":"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0","Type":"ContainerStarted","Data":"8b264ce6e9d00d0aee1a8a59ab390c13f1b958e47429ea65fe9d1e7767ab3ad7"} Oct 30 01:00:01.606586 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.606525 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" event={"ID":"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0","Type":"ContainerStarted","Data":"a80522583d5e296c38d08d0e5188276db6756974bad9c32e95b2927b84136046"} Oct 30 01:00:01.608028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.608007 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" event={"ID":"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5","Type":"ContainerStarted","Data":"e255e922fdf3aec255c4028546ea7a42e02596299f4fb40e1a3eb4ca53cde344"} Oct 30 01:00:01.609250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.609232 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" event={"ID":"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972","Type":"ContainerStarted","Data":"a5fba76573b912a53e8f25eb8f37a1ee4f4a52a3b68e83b742adde036a3e07a0"} Oct 30 01:00:01.610553 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.610532 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" event={"ID":"c15581be-8d44-43be-832f-323c013ba2c2","Type":"ContainerStarted","Data":"531b05b3020fd9988bc099df7e2a00065c9bd0d0a5f2f8890d14817df987e9e5"} Oct 30 01:00:01.611877 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.611847 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" event={"ID":"fd3e6997-7ad8-4065-96ea-78928efa2227","Type":"ContainerStarted","Data":"1b0e74ab5346ce76745ae9e80d33261bd00ad0d32cbc3a614576d6fefee7a90c"} Oct 30 01:00:01.643582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:01.643540 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" podStartSLOduration=1.643528882 podStartE2EDuration="1.643528882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:01.642106052 +0000 UTC m=+1539.554233192" watchObservedRunningTime="2026-10-30 01:00:01.643528882 +0000 UTC m=+1539.555656021" Oct 30 01:00:02.621720 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.621673 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" event={"ID":"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972","Type":"ContainerStarted","Data":"d002c7911649f173122714d08ec23bc4aad64105eb0588ed6228c66ed1d9d8c2"} Oct 30 01:00:02.623498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.623470 4517 generic.go:356] "Generic (PLEG): container finished" podID="c15581be-8d44-43be-832f-323c013ba2c2" containerID="b42730864f01af56e9e19f2f1e010a3497de72f6f6d7cedbb1b9280f5586529b" exitCode=60 Oct 30 01:00:02.623635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.623530 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" event={"ID":"c15581be-8d44-43be-832f-323c013ba2c2","Type":"ContainerDied","Data":"b42730864f01af56e9e19f2f1e010a3497de72f6f6d7cedbb1b9280f5586529b"} Oct 30 01:00:02.623812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.623795 4517 scope.go:122] "RemoveContainer" containerID="b42730864f01af56e9e19f2f1e010a3497de72f6f6d7cedbb1b9280f5586529b" Oct 30 01:00:02.625238 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.625212 4517 generic.go:356] "Generic (PLEG): container finished" podID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerID="71b341918e2294b5e1ec08ec65ddb8158eb4b499b4668c3fa2346e0725131f71" exitCode=60 Oct 30 01:00:02.625364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.625252 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" event={"ID":"fd3e6997-7ad8-4065-96ea-78928efa2227","Type":"ContainerDied","Data":"71b341918e2294b5e1ec08ec65ddb8158eb4b499b4668c3fa2346e0725131f71"} Oct 30 01:00:02.625596 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.625554 4517 scope.go:122] "RemoveContainer" containerID="71b341918e2294b5e1ec08ec65ddb8158eb4b499b4668c3fa2346e0725131f71" Oct 30 01:00:02.627335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.627284 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" event={"ID":"113b92e9-3769-4c4d-9a1d-e4f81277c68f","Type":"ContainerStarted","Data":"c011b037c29452af7f1682bc26c5cf0cc11f187f8fccc93c222a5395efc402ce"} Oct 30 01:00:02.628743 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.628727 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" event={"ID":"5696d68c-2f3d-4a6a-af56-7228baf535af","Type":"ContainerStarted","Data":"1e89e2dfdffd5ecca06bc0d14129a6580b30b3776135f77f6f697ee181b5a675"} Oct 30 01:00:02.630341 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.630322 4517 generic.go:356] "Generic (PLEG): container finished" podID="1687a186-3e99-47e3-8b70-d131495afade" containerID="53a6f1112cad549c91ec11d5f0a425c1859476ba83819a460824c5fe3d0099bc" exitCode=60 Oct 30 01:00:02.630421 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.630401 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" event={"ID":"1687a186-3e99-47e3-8b70-d131495afade","Type":"ContainerDied","Data":"53a6f1112cad549c91ec11d5f0a425c1859476ba83819a460824c5fe3d0099bc"} Oct 30 01:00:02.630618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.630601 4517 scope.go:122] "RemoveContainer" containerID="53a6f1112cad549c91ec11d5f0a425c1859476ba83819a460824c5fe3d0099bc" Oct 30 01:00:02.632066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.632041 4517 generic.go:356] "Generic (PLEG): container finished" podID="5861cf21-9842-4a34-884d-85ce63b421ee" containerID="f992a0637b7f32f38b41898a4d4e09d034120a311b2f3c3401bfc1ea0578ec82" exitCode=60 Oct 30 01:00:02.632144 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.632118 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" event={"ID":"5861cf21-9842-4a34-884d-85ce63b421ee","Type":"ContainerDied","Data":"f992a0637b7f32f38b41898a4d4e09d034120a311b2f3c3401bfc1ea0578ec82"} Oct 30 01:00:02.632284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.632269 4517 scope.go:122] "RemoveContainer" containerID="f992a0637b7f32f38b41898a4d4e09d034120a311b2f3c3401bfc1ea0578ec82" Oct 30 01:00:02.633842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.633823 4517 generic.go:356] "Generic (PLEG): container finished" podID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerID="8b264ce6e9d00d0aee1a8a59ab390c13f1b958e47429ea65fe9d1e7767ab3ad7" exitCode=60 Oct 30 01:00:02.633921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.633902 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" event={"ID":"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0","Type":"ContainerDied","Data":"8b264ce6e9d00d0aee1a8a59ab390c13f1b958e47429ea65fe9d1e7767ab3ad7"} Oct 30 01:00:02.634110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.634093 4517 scope.go:122] "RemoveContainer" containerID="8b264ce6e9d00d0aee1a8a59ab390c13f1b958e47429ea65fe9d1e7767ab3ad7" Oct 30 01:00:02.635640 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.635622 4517 generic.go:356] "Generic (PLEG): container finished" podID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerID="c74cd1600871a6d35185ecc574f739a456ff6ce9785f26af597cfaea6e6dbb8d" exitCode=7 Oct 30 01:00:02.635739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.635661 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" event={"ID":"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5","Type":"ContainerDied","Data":"c74cd1600871a6d35185ecc574f739a456ff6ce9785f26af597cfaea6e6dbb8d"} Oct 30 01:00:02.635992 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.635978 4517 scope.go:122] "RemoveContainer" containerID="c74cd1600871a6d35185ecc574f739a456ff6ce9785f26af597cfaea6e6dbb8d" Oct 30 01:00:02.662310 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.662265 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" podStartSLOduration=2.662248345 podStartE2EDuration="2.662248345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.6573166 +0000 UTC m=+1540.569443752" watchObservedRunningTime="2026-10-30 01:00:02.662248345 +0000 UTC m=+1540.574375490" Oct 30 01:00:02.731947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.731924 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:02.886179 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.885678 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" podStartSLOduration=2.88565819 podStartE2EDuration="2.88565819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.884092571 +0000 UTC m=+1540.796219711" watchObservedRunningTime="2026-10-30 01:00:02.88565819 +0000 UTC m=+1540.797785331" Oct 30 01:00:02.931121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:02.927146 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" podStartSLOduration=2.927128688 podStartE2EDuration="2.927128688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.925980805 +0000 UTC m=+1540.838107967" watchObservedRunningTime="2026-10-30 01:00:02.927128688 +0000 UTC m=+1540.839255830" Oct 30 01:00:03.002957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.002923 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:03.159783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.155697 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:03.738128 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.738097 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" event={"ID":"1687a186-3e99-47e3-8b70-d131495afade","Type":"ContainerStarted","Data":"c2121c6172a53bb8cd7d9de03cc4703c3c5a5fedff764f1737f9c1e1da6656e5"} Oct 30 01:00:03.743428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.743397 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" event={"ID":"5861cf21-9842-4a34-884d-85ce63b421ee","Type":"ContainerStarted","Data":"dcebaa9a031496a54fb8bcf27af578cc9122df2563568e0b66fb548c50a07eba"} Oct 30 01:00:03.744270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.744246 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" event={"ID":"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0","Type":"ContainerStarted","Data":"f862896b1c462e788212b2868bc0858855ca73ab26a89e5266f5b5ca1c83614f"} Oct 30 01:00:03.747066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.746561 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" event={"ID":"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5","Type":"ContainerStarted","Data":"c8963d5f3c3080cfd363567dcc8e1f9088455fb1bff2054ba117bc25852220b1"} Oct 30 01:00:03.748805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.748783 4517 generic.go:356] "Generic (PLEG): container finished" podID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerID="d002c7911649f173122714d08ec23bc4aad64105eb0588ed6228c66ed1d9d8c2" exitCode=60 Oct 30 01:00:03.748988 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.748972 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" event={"ID":"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972","Type":"ContainerDied","Data":"d002c7911649f173122714d08ec23bc4aad64105eb0588ed6228c66ed1d9d8c2"} Oct 30 01:00:03.749351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.749336 4517 scope.go:122] "RemoveContainer" containerID="d002c7911649f173122714d08ec23bc4aad64105eb0588ed6228c66ed1d9d8c2" Oct 30 01:00:03.752824 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.752802 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" event={"ID":"c15581be-8d44-43be-832f-323c013ba2c2","Type":"ContainerStarted","Data":"7156e49395ccdb75db45e1b046f8e3d8e34a6bb25f4b99a10bd9d1f2028f5898"} Oct 30 01:00:03.755623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.755600 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" event={"ID":"fd3e6997-7ad8-4065-96ea-78928efa2227","Type":"ContainerStarted","Data":"ab6810c0001bb61dcefce39434f390943cb2deceff9f63f2dc1dc4cac363fb78"} Oct 30 01:00:03.757829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.757805 4517 generic.go:356] "Generic (PLEG): container finished" podID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerID="c011b037c29452af7f1682bc26c5cf0cc11f187f8fccc93c222a5395efc402ce" exitCode=60 Oct 30 01:00:03.757938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.757902 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" event={"ID":"113b92e9-3769-4c4d-9a1d-e4f81277c68f","Type":"ContainerDied","Data":"c011b037c29452af7f1682bc26c5cf0cc11f187f8fccc93c222a5395efc402ce"} Oct 30 01:00:03.758220 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.758201 4517 scope.go:122] "RemoveContainer" containerID="c011b037c29452af7f1682bc26c5cf0cc11f187f8fccc93c222a5395efc402ce" Oct 30 01:00:03.761919 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.761889 4517 generic.go:356] "Generic (PLEG): container finished" podID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerID="1e89e2dfdffd5ecca06bc0d14129a6580b30b3776135f77f6f697ee181b5a675" exitCode=60 Oct 30 01:00:03.762053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.761923 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" event={"ID":"5696d68c-2f3d-4a6a-af56-7228baf535af","Type":"ContainerDied","Data":"1e89e2dfdffd5ecca06bc0d14129a6580b30b3776135f77f6f697ee181b5a675"} Oct 30 01:00:03.762281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.762216 4517 scope.go:122] "RemoveContainer" containerID="1e89e2dfdffd5ecca06bc0d14129a6580b30b3776135f77f6f697ee181b5a675" Oct 30 01:00:03.790257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.790202 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" podStartSLOduration=3.790184947 podStartE2EDuration="3.790184947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.774698174 +0000 UTC m=+1541.686825317" watchObservedRunningTime="2026-10-30 01:00:03.790184947 +0000 UTC m=+1541.702312087" Oct 30 01:00:03.817826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.817772 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" podStartSLOduration=3.817754437 podStartE2EDuration="3.817754437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.812141825 +0000 UTC m=+1541.724268967" watchObservedRunningTime="2026-10-30 01:00:03.817754437 +0000 UTC m=+1541.729881576" Oct 30 01:00:03.890827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.890769 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" podStartSLOduration=3.890750391 podStartE2EDuration="3.890750391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.887283777 +0000 UTC m=+1541.799410932" watchObservedRunningTime="2026-10-30 01:00:03.890750391 +0000 UTC m=+1541.802877533" Oct 30 01:00:03.926071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:03.926030 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" podStartSLOduration=3.926017674 podStartE2EDuration="3.926017674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.922436318 +0000 UTC m=+1541.834563457" watchObservedRunningTime="2026-10-30 01:00:03.926017674 +0000 UTC m=+1541.838144813" Oct 30 01:00:04.000491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.000400 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" podStartSLOduration=4.000383956 podStartE2EDuration="4.000383956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.999320137 +0000 UTC m=+1541.911447280" watchObservedRunningTime="2026-10-30 01:00:04.000383956 +0000 UTC m=+1541.912511142" Oct 30 01:00:04.409951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.409907 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:04.820765 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.820064 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" event={"ID":"113b92e9-3769-4c4d-9a1d-e4f81277c68f","Type":"ContainerStarted","Data":"472e88941a6bcf018729d5091410fd13fcbcd5ee9048a02bfcdb8296b31c7759"} Oct 30 01:00:04.834731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.826167 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" event={"ID":"5696d68c-2f3d-4a6a-af56-7228baf535af","Type":"ContainerStarted","Data":"6825f8b02e51bb785705ac4b05a95577e21a053cdc7037c29d908f3a18d2e3e3"} Oct 30 01:00:04.834731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.829882 4517 generic.go:356] "Generic (PLEG): container finished" podID="1687a186-3e99-47e3-8b70-d131495afade" containerID="c2121c6172a53bb8cd7d9de03cc4703c3c5a5fedff764f1737f9c1e1da6656e5" exitCode=60 Oct 30 01:00:04.834731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.829958 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" event={"ID":"1687a186-3e99-47e3-8b70-d131495afade","Type":"ContainerDied","Data":"c2121c6172a53bb8cd7d9de03cc4703c3c5a5fedff764f1737f9c1e1da6656e5"} Oct 30 01:00:04.834731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.829999 4517 scope.go:122] "RemoveContainer" containerID="53a6f1112cad549c91ec11d5f0a425c1859476ba83819a460824c5fe3d0099bc" Oct 30 01:00:04.834731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.830454 4517 scope.go:122] "RemoveContainer" containerID="c2121c6172a53bb8cd7d9de03cc4703c3c5a5fedff764f1737f9c1e1da6656e5" Oct 30 01:00:04.834731 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:04.830795 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm_odh-ai-gateway-infra(1687a186-3e99-47e3-8b70-d131495afade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" podUID="1687a186-3e99-47e3-8b70-d131495afade" Oct 30 01:00:04.841772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.836487 4517 generic.go:356] "Generic (PLEG): container finished" podID="5861cf21-9842-4a34-884d-85ce63b421ee" containerID="dcebaa9a031496a54fb8bcf27af578cc9122df2563568e0b66fb548c50a07eba" exitCode=60 Oct 30 01:00:04.841772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.836560 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" event={"ID":"5861cf21-9842-4a34-884d-85ce63b421ee","Type":"ContainerDied","Data":"dcebaa9a031496a54fb8bcf27af578cc9122df2563568e0b66fb548c50a07eba"} Oct 30 01:00:04.841772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.837108 4517 scope.go:122] "RemoveContainer" containerID="dcebaa9a031496a54fb8bcf27af578cc9122df2563568e0b66fb548c50a07eba" Oct 30 01:00:04.841772 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:04.837453 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk_odh-ai-gateway-infra(5861cf21-9842-4a34-884d-85ce63b421ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" Oct 30 01:00:04.841772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.839459 4517 generic.go:356] "Generic (PLEG): container finished" podID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerID="f862896b1c462e788212b2868bc0858855ca73ab26a89e5266f5b5ca1c83614f" exitCode=60 Oct 30 01:00:04.841772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.839977 4517 scope.go:122] "RemoveContainer" containerID="f862896b1c462e788212b2868bc0858855ca73ab26a89e5266f5b5ca1c83614f" Oct 30 01:00:04.841772 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:04.840298 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc_odh-ai-gateway-infra(8eedeef4-62fc-48d9-8eca-89f79ac5e4b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" Oct 30 01:00:04.841772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.840676 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" event={"ID":"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0","Type":"ContainerDied","Data":"f862896b1c462e788212b2868bc0858855ca73ab26a89e5266f5b5ca1c83614f"} Oct 30 01:00:04.848964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.848941 4517 scope.go:122] "RemoveContainer" containerID="f992a0637b7f32f38b41898a4d4e09d034120a311b2f3c3401bfc1ea0578ec82" Oct 30 01:00:04.850819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.850794 4517 generic.go:356] "Generic (PLEG): container finished" podID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerID="c8963d5f3c3080cfd363567dcc8e1f9088455fb1bff2054ba117bc25852220b1" exitCode=60 Oct 30 01:00:04.855673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.851306 4517 scope.go:122] "RemoveContainer" containerID="c8963d5f3c3080cfd363567dcc8e1f9088455fb1bff2054ba117bc25852220b1" Oct 30 01:00:04.855673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.851344 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" event={"ID":"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5","Type":"ContainerDied","Data":"c8963d5f3c3080cfd363567dcc8e1f9088455fb1bff2054ba117bc25852220b1"} Oct 30 01:00:04.855673 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:04.852438 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z_odh-ai-gateway-infra(6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" Oct 30 01:00:04.855673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.854507 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" event={"ID":"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972","Type":"ContainerStarted","Data":"dd7e43bd89c646501ed7b8bb56860d1aba6049b7b1eb028e0d3704b9a024da4e"} Oct 30 01:00:04.862925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.862890 4517 generic.go:356] "Generic (PLEG): container finished" podID="c15581be-8d44-43be-832f-323c013ba2c2" containerID="7156e49395ccdb75db45e1b046f8e3d8e34a6bb25f4b99a10bd9d1f2028f5898" exitCode=60 Oct 30 01:00:04.863057 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.862992 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" event={"ID":"c15581be-8d44-43be-832f-323c013ba2c2","Type":"ContainerDied","Data":"7156e49395ccdb75db45e1b046f8e3d8e34a6bb25f4b99a10bd9d1f2028f5898"} Oct 30 01:00:04.863489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.863465 4517 scope.go:122] "RemoveContainer" containerID="7156e49395ccdb75db45e1b046f8e3d8e34a6bb25f4b99a10bd9d1f2028f5898" Oct 30 01:00:04.864880 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:04.863868 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz_odh-ai-gateway-infra(c15581be-8d44-43be-832f-323c013ba2c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" podUID="c15581be-8d44-43be-832f-323c013ba2c2" Oct 30 01:00:04.866914 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.866529 4517 generic.go:356] "Generic (PLEG): container finished" podID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerID="ab6810c0001bb61dcefce39434f390943cb2deceff9f63f2dc1dc4cac363fb78" exitCode=60 Oct 30 01:00:04.866914 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.866722 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" event={"ID":"fd3e6997-7ad8-4065-96ea-78928efa2227","Type":"ContainerDied","Data":"ab6810c0001bb61dcefce39434f390943cb2deceff9f63f2dc1dc4cac363fb78"} Oct 30 01:00:04.868900 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.867101 4517 scope.go:122] "RemoveContainer" containerID="ab6810c0001bb61dcefce39434f390943cb2deceff9f63f2dc1dc4cac363fb78" Oct 30 01:00:04.868900 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:04.867382 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v_odh-ai-gateway-infra(fd3e6997-7ad8-4065-96ea-78928efa2227)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" Oct 30 01:00:04.883274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.883246 4517 scope.go:122] "RemoveContainer" containerID="8b264ce6e9d00d0aee1a8a59ab390c13f1b958e47429ea65fe9d1e7767ab3ad7" Oct 30 01:00:04.933488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.933466 4517 scope.go:122] "RemoveContainer" containerID="c74cd1600871a6d35185ecc574f739a456ff6ce9785f26af597cfaea6e6dbb8d" Oct 30 01:00:04.951369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:04.951346 4517 scope.go:122] "RemoveContainer" containerID="b42730864f01af56e9e19f2f1e010a3497de72f6f6d7cedbb1b9280f5586529b" Oct 30 01:00:05.066059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.066028 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:05.108146 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.108124 4517 scope.go:122] "RemoveContainer" containerID="71b341918e2294b5e1ec08ec65ddb8158eb4b499b4668c3fa2346e0725131f71" Oct 30 01:00:05.407264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.407117 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:05.575790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.571273 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:05.882772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.882701 4517 scope.go:122] "RemoveContainer" containerID="dcebaa9a031496a54fb8bcf27af578cc9122df2563568e0b66fb548c50a07eba" Oct 30 01:00:05.883347 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:05.883314 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk_odh-ai-gateway-infra(5861cf21-9842-4a34-884d-85ce63b421ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" Oct 30 01:00:05.887003 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.886971 4517 scope.go:122] "RemoveContainer" containerID="f862896b1c462e788212b2868bc0858855ca73ab26a89e5266f5b5ca1c83614f" Oct 30 01:00:05.887549 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:05.887510 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc_odh-ai-gateway-infra(8eedeef4-62fc-48d9-8eca-89f79ac5e4b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" Oct 30 01:00:05.888849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.888827 4517 scope.go:122] "RemoveContainer" containerID="c8963d5f3c3080cfd363567dcc8e1f9088455fb1bff2054ba117bc25852220b1" Oct 30 01:00:05.889163 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:05.889140 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z_odh-ai-gateway-infra(6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" Oct 30 01:00:05.890693 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.890669 4517 generic.go:356] "Generic (PLEG): container finished" podID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerID="dd7e43bd89c646501ed7b8bb56860d1aba6049b7b1eb028e0d3704b9a024da4e" exitCode=60 Oct 30 01:00:05.890823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.890699 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" event={"ID":"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972","Type":"ContainerDied","Data":"dd7e43bd89c646501ed7b8bb56860d1aba6049b7b1eb028e0d3704b9a024da4e"} Oct 30 01:00:05.890890 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.890848 4517 scope.go:122] "RemoveContainer" containerID="d002c7911649f173122714d08ec23bc4aad64105eb0588ed6228c66ed1d9d8c2" Oct 30 01:00:05.891025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.891010 4517 scope.go:122] "RemoveContainer" containerID="dd7e43bd89c646501ed7b8bb56860d1aba6049b7b1eb028e0d3704b9a024da4e" Oct 30 01:00:05.891281 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:05.891264 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm_odh-ai-gateway-infra(ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" Oct 30 01:00:05.893115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.893097 4517 scope.go:122] "RemoveContainer" containerID="7156e49395ccdb75db45e1b046f8e3d8e34a6bb25f4b99a10bd9d1f2028f5898" Oct 30 01:00:05.893368 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:05.893348 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz_odh-ai-gateway-infra(c15581be-8d44-43be-832f-323c013ba2c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" podUID="c15581be-8d44-43be-832f-323c013ba2c2" Oct 30 01:00:05.894670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.894653 4517 scope.go:122] "RemoveContainer" containerID="ab6810c0001bb61dcefce39434f390943cb2deceff9f63f2dc1dc4cac363fb78" Oct 30 01:00:05.894932 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:05.894912 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v_odh-ai-gateway-infra(fd3e6997-7ad8-4065-96ea-78928efa2227)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" Oct 30 01:00:05.897454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.897428 4517 generic.go:356] "Generic (PLEG): container finished" podID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerID="472e88941a6bcf018729d5091410fd13fcbcd5ee9048a02bfcdb8296b31c7759" exitCode=60 Oct 30 01:00:05.897553 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.897531 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" event={"ID":"113b92e9-3769-4c4d-9a1d-e4f81277c68f","Type":"ContainerDied","Data":"472e88941a6bcf018729d5091410fd13fcbcd5ee9048a02bfcdb8296b31c7759"} Oct 30 01:00:05.897689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.897669 4517 scope.go:122] "RemoveContainer" containerID="472e88941a6bcf018729d5091410fd13fcbcd5ee9048a02bfcdb8296b31c7759" Oct 30 01:00:05.897954 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:05.897935 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t_odh-ai-gateway-infra(113b92e9-3769-4c4d-9a1d-e4f81277c68f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" Oct 30 01:00:05.899700 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.899681 4517 generic.go:356] "Generic (PLEG): container finished" podID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerID="6825f8b02e51bb785705ac4b05a95577e21a053cdc7037c29d908f3a18d2e3e3" exitCode=60 Oct 30 01:00:05.899700 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.899737 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" event={"ID":"5696d68c-2f3d-4a6a-af56-7228baf535af","Type":"ContainerDied","Data":"6825f8b02e51bb785705ac4b05a95577e21a053cdc7037c29d908f3a18d2e3e3"} Oct 30 01:00:05.900009 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.899978 4517 scope.go:122] "RemoveContainer" containerID="6825f8b02e51bb785705ac4b05a95577e21a053cdc7037c29d908f3a18d2e3e3" Oct 30 01:00:05.900235 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:05.900211 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt_odh-ai-gateway-infra(5696d68c-2f3d-4a6a-af56-7228baf535af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" Oct 30 01:00:05.901656 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.901640 4517 scope.go:122] "RemoveContainer" containerID="c2121c6172a53bb8cd7d9de03cc4703c3c5a5fedff764f1737f9c1e1da6656e5" Oct 30 01:00:05.901916 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:05.901895 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm_odh-ai-gateway-infra(1687a186-3e99-47e3-8b70-d131495afade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" podUID="1687a186-3e99-47e3-8b70-d131495afade" Oct 30 01:00:05.906093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.906073 4517 scope.go:122] "RemoveContainer" containerID="c011b037c29452af7f1682bc26c5cf0cc11f187f8fccc93c222a5395efc402ce" Oct 30 01:00:05.919259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.919243 4517 scope.go:122] "RemoveContainer" containerID="1e89e2dfdffd5ecca06bc0d14129a6580b30b3776135f77f6f697ee181b5a675" Oct 30 01:00:05.967605 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:05.967580 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:06.911135 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:06.911105 4517 scope.go:122] "RemoveContainer" containerID="dd7e43bd89c646501ed7b8bb56860d1aba6049b7b1eb028e0d3704b9a024da4e" Oct 30 01:00:06.911519 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:06.911474 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm_odh-ai-gateway-infra(ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" Oct 30 01:00:06.913283 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:06.913263 4517 scope.go:122] "RemoveContainer" containerID="472e88941a6bcf018729d5091410fd13fcbcd5ee9048a02bfcdb8296b31c7759" Oct 30 01:00:06.913622 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:06.913596 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t_odh-ai-gateway-infra(113b92e9-3769-4c4d-9a1d-e4f81277c68f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" Oct 30 01:00:06.914782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:06.914765 4517 scope.go:122] "RemoveContainer" containerID="6825f8b02e51bb785705ac4b05a95577e21a053cdc7037c29d908f3a18d2e3e3" Oct 30 01:00:06.915002 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:06.914974 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt_odh-ai-gateway-infra(5696d68c-2f3d-4a6a-af56-7228baf535af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" Oct 30 01:00:07.641892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:07.641852 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:07.840072 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:07.840040 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:07.955140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:07.955028 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:09.798346 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:09.798310 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:10.325170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:10.325126 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:10.451176 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:10.451115 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:11.961826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:11.961789 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:12.392341 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:12.392308 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:12.509408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:12.509356 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:12.711973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:12.711881 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:13.447955 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.447896 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" podUID="36b8e0b5-d0bf-4a50-b448-de3117133944" containerName="main" containerID="cri-o://1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113" gracePeriod=45 Oct 30 01:00:13.540956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.540922 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:13.739814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.739790 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 01:00:13.849404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.849356 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:13.877619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.877585 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-home\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-home\") pod \"36b8e0b5-d0bf-4a50-b448-de3117133944\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " Oct 30 01:00:13.877851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.877669 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-dshm\") pod \"36b8e0b5-d0bf-4a50-b448-de3117133944\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " Oct 30 01:00:13.877851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.877768 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b8e0b5-d0bf-4a50-b448-de3117133944-kube-api-access-2gkfr\" (UniqueName: \"kubernetes.io/projected/36b8e0b5-d0bf-4a50-b448-de3117133944-kube-api-access-2gkfr\") pod \"36b8e0b5-d0bf-4a50-b448-de3117133944\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " Oct 30 01:00:13.877976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.877951 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-home" pod "36b8e0b5-d0bf-4a50-b448-de3117133944" (UID: "36b8e0b5-d0bf-4a50-b448-de3117133944"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:13.878596 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.878509 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-model-cache\") pod \"36b8e0b5-d0bf-4a50-b448-de3117133944\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " Oct 30 01:00:13.878596 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.878590 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b8e0b5-d0bf-4a50-b448-de3117133944-tls-certs\" (UniqueName: \"kubernetes.io/secret/36b8e0b5-d0bf-4a50-b448-de3117133944-tls-certs\") pod \"36b8e0b5-d0bf-4a50-b448-de3117133944\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " Oct 30 01:00:13.878873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.878619 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-tmp-dir\") pod \"36b8e0b5-d0bf-4a50-b448-de3117133944\" (UID: \"36b8e0b5-d0bf-4a50-b448-de3117133944\") " Oct 30 01:00:13.878873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.878850 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-model-cache" pod "36b8e0b5-d0bf-4a50-b448-de3117133944" (UID: "36b8e0b5-d0bf-4a50-b448-de3117133944"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:13.879012 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.878960 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-tmp-dir" pod "36b8e0b5-d0bf-4a50-b448-de3117133944" (UID: "36b8e0b5-d0bf-4a50-b448-de3117133944"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:13.879460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.879421 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:13.879460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.879444 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:13.879460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.879456 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:13.880534 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.880501 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b8e0b5-d0bf-4a50-b448-de3117133944-kube-api-access-2gkfr" pod "36b8e0b5-d0bf-4a50-b448-de3117133944" (UID: "36b8e0b5-d0bf-4a50-b448-de3117133944"). InnerVolumeSpecName "kube-api-access-2gkfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:13.880534 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.880518 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-dshm" pod "36b8e0b5-d0bf-4a50-b448-de3117133944" (UID: "36b8e0b5-d0bf-4a50-b448-de3117133944"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:13.880856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.880835 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b8e0b5-d0bf-4a50-b448-de3117133944-tls-certs" pod "36b8e0b5-d0bf-4a50-b448-de3117133944" (UID: "36b8e0b5-d0bf-4a50-b448-de3117133944"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:00:13.981536 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.981435 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b8e0b5-d0bf-4a50-b448-de3117133944-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:13.981536 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.981474 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b8e0b5-d0bf-4a50-b448-de3117133944-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:13.981536 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:13.981489 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gkfr\" (UniqueName: \"kubernetes.io/projected/36b8e0b5-d0bf-4a50-b448-de3117133944-kube-api-access-2gkfr\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:14.027843 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.027288 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:14.032128 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.032099 4517 generic.go:356] "Generic (PLEG): container finished" podID="36b8e0b5-d0bf-4a50-b448-de3117133944" containerID="1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113" exitCode=0 Oct 30 01:00:14.032287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.032164 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" Oct 30 01:00:14.032342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.032324 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" event={"ID":"36b8e0b5-d0bf-4a50-b448-de3117133944","Type":"ContainerDied","Data":"1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113"} Oct 30 01:00:14.032405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.032365 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt" event={"ID":"36b8e0b5-d0bf-4a50-b448-de3117133944","Type":"ContainerDied","Data":"1bde5297f9e0ebd82e3e95b0713f8280f87058fe773ab178ef2d06e668520c14"} Oct 30 01:00:14.032405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.032392 4517 scope.go:122] "RemoveContainer" containerID="1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113" Oct 30 01:00:14.058019 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.057991 4517 scope.go:122] "RemoveContainer" containerID="1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113" Oct 30 01:00:14.058598 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:14.058571 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113\": container with ID starting with 1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113 not found: ID does not exist" containerID="1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113" Oct 30 01:00:14.058776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.058745 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113"} err="failed to get container status \"1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113\": rpc error: code = NotFound desc = could not find container \"1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113\": container with ID starting with 1c053f5484f635459b01a9869dcaf5b763ce5f0a193249dbd6b89567f008b113 not found: ID does not exist" Oct 30 01:00:14.170889 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.170851 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt"] Oct 30 01:00:14.217734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.211940 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d8090513/e2e-lifecycle-model-d8090513-kserve-97bfc5f7c-q9qbt"] Oct 30 01:00:14.288785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.277881 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b8e0b5-d0bf-4a50-b448-de3117133944" path="/var/lib/kubelet/pods/36b8e0b5-d0bf-4a50-b448-de3117133944/volumes" Oct 30 01:00:14.412732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:14.399585 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:15.228171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:15.228136 4517 scope.go:122] "RemoveContainer" containerID="f862896b1c462e788212b2868bc0858855ca73ab26a89e5266f5b5ca1c83614f" Oct 30 01:00:15.292550 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:15.292519 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:15.579069 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:15.579035 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:16.049928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:16.049897 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" event={"ID":"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0","Type":"ContainerStarted","Data":"b7545272d5815669c0f007593e1d9f58b5a3998fe58c4e4d1c2e1bc479b0df43"} Oct 30 01:00:16.227834 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:16.227801 4517 scope.go:122] "RemoveContainer" containerID="7156e49395ccdb75db45e1b046f8e3d8e34a6bb25f4b99a10bd9d1f2028f5898" Oct 30 01:00:16.228217 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:16.228192 4517 scope.go:122] "RemoveContainer" containerID="c2121c6172a53bb8cd7d9de03cc4703c3c5a5fedff764f1737f9c1e1da6656e5" Oct 30 01:00:16.228622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:16.228338 4517 scope.go:122] "RemoveContainer" containerID="472e88941a6bcf018729d5091410fd13fcbcd5ee9048a02bfcdb8296b31c7759" Oct 30 01:00:16.228622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:16.228429 4517 scope.go:122] "RemoveContainer" containerID="dcebaa9a031496a54fb8bcf27af578cc9122df2563568e0b66fb548c50a07eba" Oct 30 01:00:17.059145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.059113 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" event={"ID":"113b92e9-3769-4c4d-9a1d-e4f81277c68f","Type":"ContainerStarted","Data":"a7cb691ce52ac117b390760e112b982ae4a6ab97e831c35f1078e70b335593cc"} Oct 30 01:00:17.062479 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.062448 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" event={"ID":"1687a186-3e99-47e3-8b70-d131495afade","Type":"ContainerStarted","Data":"526793175ebaf1a6647da2cca32756d7ad6fdd1c43e9bc9746e1bcaef26939fc"} Oct 30 01:00:17.064621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.064597 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" event={"ID":"5861cf21-9842-4a34-884d-85ce63b421ee","Type":"ContainerStarted","Data":"0b9c7e9b7c968be9a96b104c40fa6086352b3c51ea9cd71e364ee63430885c96"} Oct 30 01:00:17.067494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.067468 4517 generic.go:356] "Generic (PLEG): container finished" podID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerID="b7545272d5815669c0f007593e1d9f58b5a3998fe58c4e4d1c2e1bc479b0df43" exitCode=60 Oct 30 01:00:17.067494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.067487 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" event={"ID":"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0","Type":"ContainerDied","Data":"b7545272d5815669c0f007593e1d9f58b5a3998fe58c4e4d1c2e1bc479b0df43"} Oct 30 01:00:17.067671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.067538 4517 scope.go:122] "RemoveContainer" containerID="f862896b1c462e788212b2868bc0858855ca73ab26a89e5266f5b5ca1c83614f" Oct 30 01:00:17.067888 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.067869 4517 scope.go:122] "RemoveContainer" containerID="b7545272d5815669c0f007593e1d9f58b5a3998fe58c4e4d1c2e1bc479b0df43" Oct 30 01:00:17.068214 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:17.068192 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc_odh-ai-gateway-infra(8eedeef4-62fc-48d9-8eca-89f79ac5e4b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" Oct 30 01:00:17.069784 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.069760 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" event={"ID":"c15581be-8d44-43be-832f-323c013ba2c2","Type":"ContainerStarted","Data":"55f0f1525f772ffe90554855e296ef68b37a079102164a591249737475d8aab5"} Oct 30 01:00:17.139722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.139682 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc"] Oct 30 01:00:17.228753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.228626 4517 scope.go:122] "RemoveContainer" containerID="6825f8b02e51bb785705ac4b05a95577e21a053cdc7037c29d908f3a18d2e3e3" Oct 30 01:00:17.229211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.228750 4517 scope.go:122] "RemoveContainer" containerID="c8963d5f3c3080cfd363567dcc8e1f9088455fb1bff2054ba117bc25852220b1" Oct 30 01:00:17.229211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.228814 4517 scope.go:122] "RemoveContainer" containerID="ab6810c0001bb61dcefce39434f390943cb2deceff9f63f2dc1dc4cac363fb78" Oct 30 01:00:17.504428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.504338 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:17.895054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:17.895019 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:18.027961 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.027874 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:18.089769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.086405 4517 generic.go:356] "Generic (PLEG): container finished" podID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerID="a7cb691ce52ac117b390760e112b982ae4a6ab97e831c35f1078e70b335593cc" exitCode=60 Oct 30 01:00:18.089769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.086497 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" event={"ID":"113b92e9-3769-4c4d-9a1d-e4f81277c68f","Type":"ContainerDied","Data":"a7cb691ce52ac117b390760e112b982ae4a6ab97e831c35f1078e70b335593cc"} Oct 30 01:00:18.089769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.086552 4517 scope.go:122] "RemoveContainer" containerID="472e88941a6bcf018729d5091410fd13fcbcd5ee9048a02bfcdb8296b31c7759" Oct 30 01:00:18.089769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.087076 4517 scope.go:122] "RemoveContainer" containerID="a7cb691ce52ac117b390760e112b982ae4a6ab97e831c35f1078e70b335593cc" Oct 30 01:00:18.089769 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:18.087450 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t_odh-ai-gateway-infra(113b92e9-3769-4c4d-9a1d-e4f81277c68f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" Oct 30 01:00:18.093077 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.091043 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" event={"ID":"5696d68c-2f3d-4a6a-af56-7228baf535af","Type":"ContainerStarted","Data":"f30801d0a25dd1b7327cb2bba12c67505ceae96e85a772eb977938433e570d91"} Oct 30 01:00:18.094503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.094461 4517 generic.go:356] "Generic (PLEG): container finished" podID="1687a186-3e99-47e3-8b70-d131495afade" containerID="526793175ebaf1a6647da2cca32756d7ad6fdd1c43e9bc9746e1bcaef26939fc" exitCode=60 Oct 30 01:00:18.094699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.094557 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" event={"ID":"1687a186-3e99-47e3-8b70-d131495afade","Type":"ContainerDied","Data":"526793175ebaf1a6647da2cca32756d7ad6fdd1c43e9bc9746e1bcaef26939fc"} Oct 30 01:00:18.095291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.095270 4517 scope.go:122] "RemoveContainer" containerID="526793175ebaf1a6647da2cca32756d7ad6fdd1c43e9bc9746e1bcaef26939fc" Oct 30 01:00:18.095664 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:18.095640 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm_odh-ai-gateway-infra(1687a186-3e99-47e3-8b70-d131495afade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" podUID="1687a186-3e99-47e3-8b70-d131495afade" Oct 30 01:00:18.098189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.098158 4517 generic.go:356] "Generic (PLEG): container finished" podID="5861cf21-9842-4a34-884d-85ce63b421ee" containerID="0b9c7e9b7c968be9a96b104c40fa6086352b3c51ea9cd71e364ee63430885c96" exitCode=60 Oct 30 01:00:18.098323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.098237 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" event={"ID":"5861cf21-9842-4a34-884d-85ce63b421ee","Type":"ContainerDied","Data":"0b9c7e9b7c968be9a96b104c40fa6086352b3c51ea9cd71e364ee63430885c96"} Oct 30 01:00:18.098611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.098592 4517 scope.go:122] "RemoveContainer" containerID="0b9c7e9b7c968be9a96b104c40fa6086352b3c51ea9cd71e364ee63430885c96" Oct 30 01:00:18.098944 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:18.098922 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk_odh-ai-gateway-infra(5861cf21-9842-4a34-884d-85ce63b421ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" Oct 30 01:00:18.111232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.111203 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" event={"ID":"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5","Type":"ContainerStarted","Data":"72a5280e5073485f8556949625f7cf1e7a731802475eb5762165094b059510e4"} Oct 30 01:00:18.114420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.114391 4517 generic.go:356] "Generic (PLEG): container finished" podID="c15581be-8d44-43be-832f-323c013ba2c2" containerID="55f0f1525f772ffe90554855e296ef68b37a079102164a591249737475d8aab5" exitCode=60 Oct 30 01:00:18.114534 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.114468 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" event={"ID":"c15581be-8d44-43be-832f-323c013ba2c2","Type":"ContainerDied","Data":"55f0f1525f772ffe90554855e296ef68b37a079102164a591249737475d8aab5"} Oct 30 01:00:18.114945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.114926 4517 scope.go:122] "RemoveContainer" containerID="55f0f1525f772ffe90554855e296ef68b37a079102164a591249737475d8aab5" Oct 30 01:00:18.115352 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:18.115319 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz_odh-ai-gateway-infra(c15581be-8d44-43be-832f-323c013ba2c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" podUID="c15581be-8d44-43be-832f-323c013ba2c2" Oct 30 01:00:18.117048 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.117026 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" event={"ID":"fd3e6997-7ad8-4065-96ea-78928efa2227","Type":"ContainerStarted","Data":"c42ef4e7db9780aff0cd2305702cbe6a5f830ea8f5229494cbe23fadf7cadd5d"} Oct 30 01:00:18.123480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.123462 4517 scope.go:122] "RemoveContainer" containerID="c2121c6172a53bb8cd7d9de03cc4703c3c5a5fedff764f1737f9c1e1da6656e5" Oct 30 01:00:18.162622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.162586 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t"] Oct 30 01:00:18.173974 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.173944 4517 scope.go:122] "RemoveContainer" containerID="dcebaa9a031496a54fb8bcf27af578cc9122df2563568e0b66fb548c50a07eba" Oct 30 01:00:18.191758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.188188 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz"] Oct 30 01:00:18.206947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.206316 4517 scope.go:122] "RemoveContainer" containerID="7156e49395ccdb75db45e1b046f8e3d8e34a6bb25f4b99a10bd9d1f2028f5898" Oct 30 01:00:18.270410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.270368 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk"] Oct 30 01:00:18.329965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.329936 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm"] Oct 30 01:00:18.355828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.355801 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:18.426694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.426658 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-kube-api-access-fvglr\" (UniqueName: \"kubernetes.io/projected/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-kube-api-access-fvglr\") pod \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\" (UID: \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\") " Oct 30 01:00:18.426932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.426915 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-maas-api-service-ca\") pod \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\" (UID: \"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0\") " Oct 30 01:00:18.427257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.427227 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-maas-api-service-ca" pod "8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" (UID: "8eedeef4-62fc-48d9-8eca-89f79ac5e4b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:18.427493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.427474 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.459939 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.459445 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-kube-api-access-fvglr" pod "8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" (UID: "8eedeef4-62fc-48d9-8eca-89f79ac5e4b0"). InnerVolumeSpecName "kube-api-access-fvglr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:18.529476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:18.529445 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvglr\" (UniqueName: \"kubernetes.io/projected/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0-kube-api-access-fvglr\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.127565 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.127536 4517 generic.go:356] "Generic (PLEG): container finished" podID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerID="f30801d0a25dd1b7327cb2bba12c67505ceae96e85a772eb977938433e570d91" exitCode=60 Oct 30 01:00:19.127752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.127612 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" event={"ID":"5696d68c-2f3d-4a6a-af56-7228baf535af","Type":"ContainerDied","Data":"f30801d0a25dd1b7327cb2bba12c67505ceae96e85a772eb977938433e570d91"} Oct 30 01:00:19.127752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.127685 4517 scope.go:122] "RemoveContainer" containerID="6825f8b02e51bb785705ac4b05a95577e21a053cdc7037c29d908f3a18d2e3e3" Oct 30 01:00:19.128165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.128146 4517 scope.go:122] "RemoveContainer" containerID="f30801d0a25dd1b7327cb2bba12c67505ceae96e85a772eb977938433e570d91" Oct 30 01:00:19.128552 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:19.128526 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt_odh-ai-gateway-infra(5696d68c-2f3d-4a6a-af56-7228baf535af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" Oct 30 01:00:19.133816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.133768 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" Oct 30 01:00:19.135084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.134506 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc" event={"ID":"8eedeef4-62fc-48d9-8eca-89f79ac5e4b0","Type":"ContainerDied","Data":"a80522583d5e296c38d08d0e5188276db6756974bad9c32e95b2927b84136046"} Oct 30 01:00:19.137848 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.137821 4517 generic.go:356] "Generic (PLEG): container finished" podID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerID="72a5280e5073485f8556949625f7cf1e7a731802475eb5762165094b059510e4" exitCode=60 Oct 30 01:00:19.137936 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.137915 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" event={"ID":"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5","Type":"ContainerDied","Data":"72a5280e5073485f8556949625f7cf1e7a731802475eb5762165094b059510e4"} Oct 30 01:00:19.138330 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.138303 4517 scope.go:122] "RemoveContainer" containerID="72a5280e5073485f8556949625f7cf1e7a731802475eb5762165094b059510e4" Oct 30 01:00:19.138648 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:19.138624 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z_odh-ai-gateway-infra(6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" Oct 30 01:00:19.143361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.143342 4517 scope.go:122] "RemoveContainer" containerID="b7545272d5815669c0f007593e1d9f58b5a3998fe58c4e4d1c2e1bc479b0df43" Oct 30 01:00:19.145130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.145108 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" event={"ID":"fd3e6997-7ad8-4065-96ea-78928efa2227","Type":"ContainerDied","Data":"c42ef4e7db9780aff0cd2305702cbe6a5f830ea8f5229494cbe23fadf7cadd5d"} Oct 30 01:00:19.145503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.145487 4517 scope.go:122] "RemoveContainer" containerID="c42ef4e7db9780aff0cd2305702cbe6a5f830ea8f5229494cbe23fadf7cadd5d" Oct 30 01:00:19.145900 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:19.145873 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v_odh-ai-gateway-infra(fd3e6997-7ad8-4065-96ea-78928efa2227)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" Oct 30 01:00:19.146005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.145949 4517 generic.go:356] "Generic (PLEG): container finished" podID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerID="c42ef4e7db9780aff0cd2305702cbe6a5f830ea8f5229494cbe23fadf7cadd5d" exitCode=60 Oct 30 01:00:19.183420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.183395 4517 scope.go:122] "RemoveContainer" containerID="c8963d5f3c3080cfd363567dcc8e1f9088455fb1bff2054ba117bc25852220b1" Oct 30 01:00:19.211129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.211101 4517 scope.go:122] "RemoveContainer" containerID="ab6810c0001bb61dcefce39434f390943cb2deceff9f63f2dc1dc4cac363fb78" Oct 30 01:00:19.235239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.229835 4517 scope.go:122] "RemoveContainer" containerID="dd7e43bd89c646501ed7b8bb56860d1aba6049b7b1eb028e0d3704b9a024da4e" Oct 30 01:00:19.278123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.276075 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt"] Oct 30 01:00:19.360597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.360561 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z"] Oct 30 01:00:19.386033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.385782 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc"] Oct 30 01:00:19.390565 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.390539 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888700-njbmc"] Oct 30 01:00:19.401979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.401925 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v"] Oct 30 01:00:19.450625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.450596 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:19.456950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.456925 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:19.539582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.539553 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:19.542427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.542399 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:19.558575 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.558532 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15581be-8d44-43be-832f-323c013ba2c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15581be-8d44-43be-832f-323c013ba2c2-maas-api-service-ca\") pod \"c15581be-8d44-43be-832f-323c013ba2c2\" (UID: \"c15581be-8d44-43be-832f-323c013ba2c2\") " Oct 30 01:00:19.558789 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.558638 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15581be-8d44-43be-832f-323c013ba2c2-kube-api-access-7x9nl\" (UniqueName: \"kubernetes.io/projected/c15581be-8d44-43be-832f-323c013ba2c2-kube-api-access-7x9nl\") pod \"c15581be-8d44-43be-832f-323c013ba2c2\" (UID: \"c15581be-8d44-43be-832f-323c013ba2c2\") " Oct 30 01:00:19.558855 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.558791 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5861cf21-9842-4a34-884d-85ce63b421ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5861cf21-9842-4a34-884d-85ce63b421ee-maas-api-service-ca\") pod \"5861cf21-9842-4a34-884d-85ce63b421ee\" (UID: \"5861cf21-9842-4a34-884d-85ce63b421ee\") " Oct 30 01:00:19.558965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.558945 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5861cf21-9842-4a34-884d-85ce63b421ee-kube-api-access-4nrjl\" (UniqueName: \"kubernetes.io/projected/5861cf21-9842-4a34-884d-85ce63b421ee-kube-api-access-4nrjl\") pod \"5861cf21-9842-4a34-884d-85ce63b421ee\" (UID: \"5861cf21-9842-4a34-884d-85ce63b421ee\") " Oct 30 01:00:19.559042 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.558966 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15581be-8d44-43be-832f-323c013ba2c2-maas-api-service-ca" pod "c15581be-8d44-43be-832f-323c013ba2c2" (UID: "c15581be-8d44-43be-832f-323c013ba2c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:19.559394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.559369 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5861cf21-9842-4a34-884d-85ce63b421ee-maas-api-service-ca" pod "5861cf21-9842-4a34-884d-85ce63b421ee" (UID: "5861cf21-9842-4a34-884d-85ce63b421ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:19.560248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.559784 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15581be-8d44-43be-832f-323c013ba2c2-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.560248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.559805 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5861cf21-9842-4a34-884d-85ce63b421ee-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.563284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.563263 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5861cf21-9842-4a34-884d-85ce63b421ee-kube-api-access-4nrjl" pod "5861cf21-9842-4a34-884d-85ce63b421ee" (UID: "5861cf21-9842-4a34-884d-85ce63b421ee"). InnerVolumeSpecName "kube-api-access-4nrjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:19.571275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.570072 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:19.576386 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.575853 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15581be-8d44-43be-832f-323c013ba2c2-kube-api-access-7x9nl" pod "c15581be-8d44-43be-832f-323c013ba2c2" (UID: "c15581be-8d44-43be-832f-323c013ba2c2"). InnerVolumeSpecName "kube-api-access-7x9nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:19.661327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.660938 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1687a186-3e99-47e3-8b70-d131495afade-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1687a186-3e99-47e3-8b70-d131495afade-maas-api-service-ca\") pod \"1687a186-3e99-47e3-8b70-d131495afade\" (UID: \"1687a186-3e99-47e3-8b70-d131495afade\") " Oct 30 01:00:19.661327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.661141 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113b92e9-3769-4c4d-9a1d-e4f81277c68f-kube-api-access-xbcpp\" (UniqueName: \"kubernetes.io/projected/113b92e9-3769-4c4d-9a1d-e4f81277c68f-kube-api-access-xbcpp\") pod \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\" (UID: \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\") " Oct 30 01:00:19.661327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.661202 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1687a186-3e99-47e3-8b70-d131495afade-kube-api-access-rgtbd\" (UniqueName: \"kubernetes.io/projected/1687a186-3e99-47e3-8b70-d131495afade-kube-api-access-rgtbd\") pod \"1687a186-3e99-47e3-8b70-d131495afade\" (UID: \"1687a186-3e99-47e3-8b70-d131495afade\") " Oct 30 01:00:19.661611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.661358 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/113b92e9-3769-4c4d-9a1d-e4f81277c68f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113b92e9-3769-4c4d-9a1d-e4f81277c68f-maas-api-service-ca\") pod \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\" (UID: \"113b92e9-3769-4c4d-9a1d-e4f81277c68f\") " Oct 30 01:00:19.662052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.662025 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nrjl\" (UniqueName: \"kubernetes.io/projected/5861cf21-9842-4a34-884d-85ce63b421ee-kube-api-access-4nrjl\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.662142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.662055 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x9nl\" (UniqueName: \"kubernetes.io/projected/c15581be-8d44-43be-832f-323c013ba2c2-kube-api-access-7x9nl\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.664884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.664838 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113b92e9-3769-4c4d-9a1d-e4f81277c68f-maas-api-service-ca" pod "113b92e9-3769-4c4d-9a1d-e4f81277c68f" (UID: "113b92e9-3769-4c4d-9a1d-e4f81277c68f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:19.665079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.665048 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1687a186-3e99-47e3-8b70-d131495afade-kube-api-access-rgtbd" pod "1687a186-3e99-47e3-8b70-d131495afade" (UID: "1687a186-3e99-47e3-8b70-d131495afade"). InnerVolumeSpecName "kube-api-access-rgtbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:19.665342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.665325 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1687a186-3e99-47e3-8b70-d131495afade-maas-api-service-ca" pod "1687a186-3e99-47e3-8b70-d131495afade" (UID: "1687a186-3e99-47e3-8b70-d131495afade"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:19.668456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.667263 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113b92e9-3769-4c4d-9a1d-e4f81277c68f-kube-api-access-xbcpp" pod "113b92e9-3769-4c4d-9a1d-e4f81277c68f" (UID: "113b92e9-3769-4c4d-9a1d-e4f81277c68f"). InnerVolumeSpecName "kube-api-access-xbcpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:19.703728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.703603 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:19.763658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.763625 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1687a186-3e99-47e3-8b70-d131495afade-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.763823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.763662 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbcpp\" (UniqueName: \"kubernetes.io/projected/113b92e9-3769-4c4d-9a1d-e4f81277c68f-kube-api-access-xbcpp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.763823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.763678 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgtbd\" (UniqueName: \"kubernetes.io/projected/1687a186-3e99-47e3-8b70-d131495afade-kube-api-access-rgtbd\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.763823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:19.763695 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/113b92e9-3769-4c4d-9a1d-e4f81277c68f-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:20.156196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.156155 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" event={"ID":"113b92e9-3769-4c4d-9a1d-e4f81277c68f","Type":"ContainerDied","Data":"b5b92400b462389be58a87713081654fcdabc4b6d0fd9ea5b95ed68e90749e1a"} Oct 30 01:00:20.156390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.156197 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t" Oct 30 01:00:20.156390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.156241 4517 scope.go:122] "RemoveContainer" containerID="a7cb691ce52ac117b390760e112b982ae4a6ab97e831c35f1078e70b335593cc" Oct 30 01:00:20.160520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.160489 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" event={"ID":"1687a186-3e99-47e3-8b70-d131495afade","Type":"ContainerDied","Data":"f691a735e5cbd98bd65a3f59a540952798e5ef4a72ecd55cf799a0ee344786cb"} Oct 30 01:00:20.160679 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.160595 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm" Oct 30 01:00:20.168088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.166099 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" event={"ID":"5861cf21-9842-4a34-884d-85ce63b421ee","Type":"ContainerDied","Data":"a02b326faaef94fff12ddc096560969bcdffc6befa3f7c0cd05d80acb93d9724"} Oct 30 01:00:20.168088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.166164 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk" Oct 30 01:00:20.179222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.178337 4517 scope.go:122] "RemoveContainer" containerID="526793175ebaf1a6647da2cca32756d7ad6fdd1c43e9bc9746e1bcaef26939fc" Oct 30 01:00:20.219599 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.219562 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" event={"ID":"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972","Type":"ContainerStarted","Data":"60f7c269651bbc4ed3138a5fb52055b9fcfb27d74a93a499b8348691af10f31f"} Oct 30 01:00:20.234165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.234134 4517 scope.go:122] "RemoveContainer" containerID="0b9c7e9b7c968be9a96b104c40fa6086352b3c51ea9cd71e364ee63430885c96" Oct 30 01:00:20.234294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.234277 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" Oct 30 01:00:20.253594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.253559 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" path="/var/lib/kubelet/pods/8eedeef4-62fc-48d9-8eca-89f79ac5e4b0/volumes" Oct 30 01:00:20.254483 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.254456 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz" event={"ID":"c15581be-8d44-43be-832f-323c013ba2c2","Type":"ContainerDied","Data":"531b05b3020fd9988bc099df7e2a00065c9bd0d0a5f2f8890d14817df987e9e5"} Oct 30 01:00:20.265158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.265138 4517 scope.go:122] "RemoveContainer" containerID="55f0f1525f772ffe90554855e296ef68b37a079102164a591249737475d8aab5" Oct 30 01:00:20.288452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.288387 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk"] Oct 30 01:00:20.294237 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.294199 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888700-frqhk"] Oct 30 01:00:20.325605 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.325571 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm"] Oct 30 01:00:20.333653 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.333213 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888700-h8rjm"] Oct 30 01:00:20.363086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.362543 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t"] Oct 30 01:00:20.372951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.371685 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888700-zf67t"] Oct 30 01:00:20.415696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.415071 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz"] Oct 30 01:00:20.419404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.419377 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888700-gzcpz"] Oct 30 01:00:20.488760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.485938 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:20.521814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.521791 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:20.525258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.525224 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:20.614246 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.614212 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-maas-api-service-ca\") pod \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\" (UID: \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\") " Oct 30 01:00:20.614431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.614333 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3e6997-7ad8-4065-96ea-78928efa2227-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3e6997-7ad8-4065-96ea-78928efa2227-maas-api-service-ca\") pod \"fd3e6997-7ad8-4065-96ea-78928efa2227\" (UID: \"fd3e6997-7ad8-4065-96ea-78928efa2227\") " Oct 30 01:00:20.614431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.614381 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5696d68c-2f3d-4a6a-af56-7228baf535af-kube-api-access-rzj2h\" (UniqueName: \"kubernetes.io/projected/5696d68c-2f3d-4a6a-af56-7228baf535af-kube-api-access-rzj2h\") pod \"5696d68c-2f3d-4a6a-af56-7228baf535af\" (UID: \"5696d68c-2f3d-4a6a-af56-7228baf535af\") " Oct 30 01:00:20.614568 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.614541 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-kube-api-access-7dblj\" (UniqueName: \"kubernetes.io/projected/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-kube-api-access-7dblj\") pod \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\" (UID: \"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5\") " Oct 30 01:00:20.614621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.614594 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5696d68c-2f3d-4a6a-af56-7228baf535af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696d68c-2f3d-4a6a-af56-7228baf535af-maas-api-service-ca\") pod \"5696d68c-2f3d-4a6a-af56-7228baf535af\" (UID: \"5696d68c-2f3d-4a6a-af56-7228baf535af\") " Oct 30 01:00:20.614621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.614604 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-maas-api-service-ca" pod "6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" (UID: "6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:20.614754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.614636 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3e6997-7ad8-4065-96ea-78928efa2227-kube-api-access-jw4l7\" (UniqueName: \"kubernetes.io/projected/fd3e6997-7ad8-4065-96ea-78928efa2227-kube-api-access-jw4l7\") pod \"fd3e6997-7ad8-4065-96ea-78928efa2227\" (UID: \"fd3e6997-7ad8-4065-96ea-78928efa2227\") " Oct 30 01:00:20.614754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.614689 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3e6997-7ad8-4065-96ea-78928efa2227-maas-api-service-ca" pod "fd3e6997-7ad8-4065-96ea-78928efa2227" (UID: "fd3e6997-7ad8-4065-96ea-78928efa2227"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:20.615053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.615025 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5696d68c-2f3d-4a6a-af56-7228baf535af-maas-api-service-ca" pod "5696d68c-2f3d-4a6a-af56-7228baf535af" (UID: "5696d68c-2f3d-4a6a-af56-7228baf535af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:20.615378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.615357 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696d68c-2f3d-4a6a-af56-7228baf535af-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:20.615460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.615385 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:20.615460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.615400 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3e6997-7ad8-4065-96ea-78928efa2227-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:20.616914 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.616892 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5696d68c-2f3d-4a6a-af56-7228baf535af-kube-api-access-rzj2h" pod "5696d68c-2f3d-4a6a-af56-7228baf535af" (UID: "5696d68c-2f3d-4a6a-af56-7228baf535af"). InnerVolumeSpecName "kube-api-access-rzj2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:20.616988 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.616920 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-kube-api-access-7dblj" pod "6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" (UID: "6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5"). InnerVolumeSpecName "kube-api-access-7dblj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:20.617213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.617190 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3e6997-7ad8-4065-96ea-78928efa2227-kube-api-access-jw4l7" pod "fd3e6997-7ad8-4065-96ea-78928efa2227" (UID: "fd3e6997-7ad8-4065-96ea-78928efa2227"). InnerVolumeSpecName "kube-api-access-jw4l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:20.717285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.717210 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzj2h\" (UniqueName: \"kubernetes.io/projected/5696d68c-2f3d-4a6a-af56-7228baf535af-kube-api-access-rzj2h\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:20.717285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.717237 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dblj\" (UniqueName: \"kubernetes.io/projected/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5-kube-api-access-7dblj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:20.717285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:20.717247 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw4l7\" (UniqueName: \"kubernetes.io/projected/fd3e6997-7ad8-4065-96ea-78928efa2227-kube-api-access-jw4l7\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:21.250508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.250479 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" Oct 30 01:00:21.250671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.250479 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z" event={"ID":"6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5","Type":"ContainerDied","Data":"e255e922fdf3aec255c4028546ea7a42e02596299f4fb40e1a3eb4ca53cde344"} Oct 30 01:00:21.250671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.250615 4517 scope.go:122] "RemoveContainer" containerID="72a5280e5073485f8556949625f7cf1e7a731802475eb5762165094b059510e4" Oct 30 01:00:21.253069 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.253043 4517 generic.go:356] "Generic (PLEG): container finished" podID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerID="60f7c269651bbc4ed3138a5fb52055b9fcfb27d74a93a499b8348691af10f31f" exitCode=60 Oct 30 01:00:21.253183 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.253074 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" event={"ID":"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972","Type":"ContainerDied","Data":"60f7c269651bbc4ed3138a5fb52055b9fcfb27d74a93a499b8348691af10f31f"} Oct 30 01:00:21.253580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.253558 4517 scope.go:122] "RemoveContainer" containerID="60f7c269651bbc4ed3138a5fb52055b9fcfb27d74a93a499b8348691af10f31f" Oct 30 01:00:21.253999 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:21.253978 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm_odh-ai-gateway-infra(ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" Oct 30 01:00:21.255656 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.255639 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" Oct 30 01:00:21.255803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.255682 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v" event={"ID":"fd3e6997-7ad8-4065-96ea-78928efa2227","Type":"ContainerDied","Data":"1b0e74ab5346ce76745ae9e80d33261bd00ad0d32cbc3a614576d6fefee7a90c"} Oct 30 01:00:21.258469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.258450 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" Oct 30 01:00:21.259146 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.259119 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt" event={"ID":"5696d68c-2f3d-4a6a-af56-7228baf535af","Type":"ContainerDied","Data":"27c342ac9597f924951ed59a3170a58db10cad21d919a5aae43469e9012ee0fa"} Oct 30 01:00:21.264924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.264856 4517 scope.go:122] "RemoveContainer" containerID="dd7e43bd89c646501ed7b8bb56860d1aba6049b7b1eb028e0d3704b9a024da4e" Oct 30 01:00:21.279601 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.279584 4517 scope.go:122] "RemoveContainer" containerID="c42ef4e7db9780aff0cd2305702cbe6a5f830ea8f5229494cbe23fadf7cadd5d" Oct 30 01:00:21.288590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.288566 4517 scope.go:122] "RemoveContainer" containerID="f30801d0a25dd1b7327cb2bba12c67505ceae96e85a772eb977938433e570d91" Oct 30 01:00:21.300211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.300188 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z"] Oct 30 01:00:21.305954 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.305931 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f4f9b-29888700-rvp6z"] Oct 30 01:00:21.396883 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.396849 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt"] Oct 30 01:00:21.401210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.401181 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-601aea-29888700-k49kt"] Oct 30 01:00:21.431356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.431328 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:21.432826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.432802 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v"] Oct 30 01:00:21.436120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.436096 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888700-sk44v"] Oct 30 01:00:21.469077 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:21.469034 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm"] Oct 30 01:00:22.247327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.247296 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" path="/var/lib/kubelet/pods/113b92e9-3769-4c4d-9a1d-e4f81277c68f/volumes" Oct 30 01:00:22.247755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.247741 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1687a186-3e99-47e3-8b70-d131495afade" path="/var/lib/kubelet/pods/1687a186-3e99-47e3-8b70-d131495afade/volumes" Oct 30 01:00:22.248127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.248109 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" path="/var/lib/kubelet/pods/5696d68c-2f3d-4a6a-af56-7228baf535af/volumes" Oct 30 01:00:22.248792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.248775 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" path="/var/lib/kubelet/pods/5861cf21-9842-4a34-884d-85ce63b421ee/volumes" Oct 30 01:00:22.249170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.249157 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" path="/var/lib/kubelet/pods/6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5/volumes" Oct 30 01:00:22.249519 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.249507 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15581be-8d44-43be-832f-323c013ba2c2" path="/var/lib/kubelet/pods/c15581be-8d44-43be-832f-323c013ba2c2/volumes" Oct 30 01:00:22.250094 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.250078 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" path="/var/lib/kubelet/pods/fd3e6997-7ad8-4065-96ea-78928efa2227/volumes" Oct 30 01:00:22.461266 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.461240 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:22.549164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.549129 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-maas-api-service-ca\") pod \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\" (UID: \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\") " Oct 30 01:00:22.549352 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.549234 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-kube-api-access-xk9vb\" (UniqueName: \"kubernetes.io/projected/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-kube-api-access-xk9vb\") pod \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\" (UID: \"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972\") " Oct 30 01:00:22.549550 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.549525 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-maas-api-service-ca" pod "ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" (UID: "ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:22.549831 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.549815 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:22.551455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.551433 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-kube-api-access-xk9vb" pod "ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" (UID: "ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972"). InnerVolumeSpecName "kube-api-access-xk9vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:22.650855 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:22.650805 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk9vb\" (UniqueName: \"kubernetes.io/projected/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972-kube-api-access-xk9vb\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:00:23.189317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:23.189279 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:23.305881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:23.305853 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" Oct 30 01:00:23.306054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:23.305851 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm" event={"ID":"ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972","Type":"ContainerDied","Data":"a5fba76573b912a53e8f25eb8f37a1ee4f4a52a3b68e83b742adde036a3e07a0"} Oct 30 01:00:23.306054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:23.305984 4517 scope.go:122] "RemoveContainer" containerID="60f7c269651bbc4ed3138a5fb52055b9fcfb27d74a93a499b8348691af10f31f" Oct 30 01:00:23.353500 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:23.353470 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm"] Oct 30 01:00:23.365501 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:23.365464 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888700-9k8gm"] Oct 30 01:00:24.244620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:24.244582 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" path="/var/lib/kubelet/pods/ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972/volumes" Oct 30 01:00:24.865740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:24.863891 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:26.668265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:26.668227 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:28.295386 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:28.295345 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:30.760824 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:00:30.760785 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:00:30.793017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:30.792990 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:30.908204 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:30.908166 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:31.315654 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:31.315184 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:31.465289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:31.465254 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:31.602807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:31.596169 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:33.307788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:33.307445 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:33.836098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:33.836054 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:34.038502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:34.038471 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:34.364029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:34.363978 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:34.732452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:34.732366 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:35.536265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:35.536232 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:35.724081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:35.724047 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:36.086151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:36.086116 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:37.061412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:37.061370 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:37.410902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:37.410809 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:37.553285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:37.553247 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:38.088345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:38.088314 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:38.652453 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:38.652421 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:38.801189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:38.801156 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:39.187359 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:39.187327 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:40.372339 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:40.372303 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:40.521213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:40.521180 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:41.550158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:41.550125 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:43.211677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:43.211633 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:44.434506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:44.434471 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:44.935735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:44.935687 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:45.449111 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:45.449070 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:45.608736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:45.604142 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:45.794592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:45.794557 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:46.345084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:46.345042 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:46.490351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:46.490317 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:46.761659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:46.761561 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:48.866109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:48.866071 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:50.000722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:50.000681 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:50.186373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:50.186333 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:51.136575 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:51.136542 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:51.736274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:51.736243 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:51.972160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:51.972123 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:53.442994 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:53.442947 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:53.579552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:53.579502 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:55.363436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:55.363371 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:57.187250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:57.187168 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:59.245229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:59.245196 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:59.367413 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:59.367306 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:59.850247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:00:59.850174 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:00.471731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:00.468932 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:00.765415 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:01:00.765315 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:01:01.330907 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:01.330872 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:03.296971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:03.296930 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:04.220054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:04.220022 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:04.402888 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:04.402850 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:05.381900 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:05.381863 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:05.999981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:05.999951 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:06.185523 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:06.185387 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:07.480973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:07.480901 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:08.412611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:08.412574 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:08.647438 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:08.647400 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:08.815071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:08.815021 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:09.194275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:09.194195 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:11.321812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:11.321757 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:12.919365 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:12.919270 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:13.755456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:13.755380 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:14.144235 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:14.144141 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:14.557768 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:14.557695 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:14.843042 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:14.842935 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:15.517824 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:15.517771 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:16.639866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:16.639834 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:16.793296 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:16.793258 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:16.923562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:16.923474 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:17.476241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:17.476203 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:17.617273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:17.617238 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:17.896410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:17.896370 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:18.386853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:18.386607 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:18.570651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:18.570612 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:18.784341 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:18.784269 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:19.037867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:19.037780 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:19.396079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:19.395998 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:19.585819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:19.575254 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:20.462399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:20.462346 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt"] Oct 30 01:01:20.463440 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:20.462622 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" containerID="cri-o://7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b" gracePeriod=30 Oct 30 01:01:20.527381 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:20.527342 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m"] Oct 30 01:01:20.527629 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:20.527602 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" containerID="cri-o://a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b" gracePeriod=30 Oct 30 01:01:20.833852 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:20.833812 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22"] Oct 30 01:01:20.834344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:20.834118 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerName="istio-proxy" containerID="cri-o://adcee1d7d371cc3f3d8183e9463d484463d3ca36ff67ea5f229d0e4c1679c447" gracePeriod=30 Oct 30 01:01:21.320661 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:21.320627 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd"] Oct 30 01:01:21.320957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:21.320893 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" containerID="cri-o://32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9" gracePeriod=30 Oct 30 01:01:21.322630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:21.322581 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:21.521353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:21.521323 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:23.822237 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:23.822190 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:24.086594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:24.086492 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:25.857683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:25.857652 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 01:01:25.926273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:25.926246 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 01:01:26.017380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.017300 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-data\") pod \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " Oct 30 01:01:26.017868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.017831 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-podinfo\") pod \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " Oct 30 01:01:26.018072 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.018057 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-token\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-token\") pod \"5022b50c-515a-4ca3-af29-ee8ce01de949\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " Oct 30 01:01:26.018257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.018226 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-credential-socket\") pod \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " Oct 30 01:01:26.018549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.018532 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-socket\") pod \"5022b50c-515a-4ca3-af29-ee8ce01de949\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " Oct 30 01:01:26.018691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.018678 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istiod-ca-cert\") pod \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " Oct 30 01:01:26.018947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.018899 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-socket" pod "5022b50c-515a-4ca3-af29-ee8ce01de949" (UID: "5022b50c-515a-4ca3-af29-ee8ce01de949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.019130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.018962 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-envoy\") pod \"5022b50c-515a-4ca3-af29-ee8ce01de949\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " Oct 30 01:01:26.019130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.019100 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-credential-socket" pod "2c81e6f4-4b34-4711-ba3f-09be57119c0a" (UID: "2c81e6f4-4b34-4711-ba3f-09be57119c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.019316 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.019272 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istiod-ca-cert" pod "2c81e6f4-4b34-4711-ba3f-09be57119c0a" (UID: "2c81e6f4-4b34-4711-ba3f-09be57119c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:26.019397 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.019372 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-credential-socket\") pod \"5022b50c-515a-4ca3-af29-ee8ce01de949\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " Oct 30 01:01:26.019737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.019635 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-credential-socket" pod "5022b50c-515a-4ca3-af29-ee8ce01de949" (UID: "5022b50c-515a-4ca3-af29-ee8ce01de949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.019814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.019696 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-envoy\") pod \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " Oct 30 01:01:26.019814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.019803 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-token\") pod \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " Oct 30 01:01:26.019923 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.019842 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-socket\") pod \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " Oct 30 01:01:26.019923 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.019892 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-data\") pod \"5022b50c-515a-4ca3-af29-ee8ce01de949\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " Oct 30 01:01:26.020028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020017 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5022b50c-515a-4ca3-af29-ee8ce01de949-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5022b50c-515a-4ca3-af29-ee8ce01de949-istiod-ca-cert\") pod \"5022b50c-515a-4ca3-af29-ee8ce01de949\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " Oct 30 01:01:26.020090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020047 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-certs\") pod \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " Oct 30 01:01:26.020145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020084 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-kube-api-access-fkwft\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-kube-api-access-fkwft\") pod \"5022b50c-515a-4ca3-af29-ee8ce01de949\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " Oct 30 01:01:26.020214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020196 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-podinfo\") pod \"5022b50c-515a-4ca3-af29-ee8ce01de949\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " Oct 30 01:01:26.020266 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020223 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-socket" pod "2c81e6f4-4b34-4711-ba3f-09be57119c0a" (UID: "2c81e6f4-4b34-4711-ba3f-09be57119c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.020320 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020272 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-certs\") pod \"5022b50c-515a-4ca3-af29-ee8ce01de949\" (UID: \"5022b50c-515a-4ca3-af29-ee8ce01de949\") " Oct 30 01:01:26.020370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020337 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-kube-api-access-l2ftn\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-kube-api-access-l2ftn\") pod \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\" (UID: \"2c81e6f4-4b34-4711-ba3f-09be57119c0a\") " Oct 30 01:01:26.020698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020674 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-certs" pod "2c81e6f4-4b34-4711-ba3f-09be57119c0a" (UID: "2c81e6f4-4b34-4711-ba3f-09be57119c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.020791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020697 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-data" pod "5022b50c-515a-4ca3-af29-ee8ce01de949" (UID: "5022b50c-515a-4ca3-af29-ee8ce01de949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.020986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.020965 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5022b50c-515a-4ca3-af29-ee8ce01de949-istiod-ca-cert" pod "5022b50c-515a-4ca3-af29-ee8ce01de949" (UID: "5022b50c-515a-4ca3-af29-ee8ce01de949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:26.021293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021271 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.021379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021299 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.021379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021314 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.021379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021327 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.021379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021340 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.021379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021353 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.021379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021366 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5022b50c-515a-4ca3-af29-ee8ce01de949-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.021379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021378 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.021768 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021489 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-certs" pod "5022b50c-515a-4ca3-af29-ee8ce01de949" (UID: "5022b50c-515a-4ca3-af29-ee8ce01de949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.021878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.021857 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-data" pod "2c81e6f4-4b34-4711-ba3f-09be57119c0a" (UID: "2c81e6f4-4b34-4711-ba3f-09be57119c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.023871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.023326 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-envoy" pod "5022b50c-515a-4ca3-af29-ee8ce01de949" (UID: "5022b50c-515a-4ca3-af29-ee8ce01de949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.023871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.023792 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:26.025151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.025097 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-kube-api-access-l2ftn" pod "2c81e6f4-4b34-4711-ba3f-09be57119c0a" (UID: "2c81e6f4-4b34-4711-ba3f-09be57119c0a"). InnerVolumeSpecName "kube-api-access-l2ftn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.031132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.030827 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-podinfo" pod "2c81e6f4-4b34-4711-ba3f-09be57119c0a" (UID: "2c81e6f4-4b34-4711-ba3f-09be57119c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:26.031132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.030888 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-kube-api-access-fkwft" pod "5022b50c-515a-4ca3-af29-ee8ce01de949" (UID: "5022b50c-515a-4ca3-af29-ee8ce01de949"). InnerVolumeSpecName "kube-api-access-fkwft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.031132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.030932 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-token" pod "5022b50c-515a-4ca3-af29-ee8ce01de949" (UID: "5022b50c-515a-4ca3-af29-ee8ce01de949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.031132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.030971 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-token" pod "2c81e6f4-4b34-4711-ba3f-09be57119c0a" (UID: "2c81e6f4-4b34-4711-ba3f-09be57119c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.031132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.031103 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-envoy" pod "2c81e6f4-4b34-4711-ba3f-09be57119c0a" (UID: "2c81e6f4-4b34-4711-ba3f-09be57119c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.033773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.032537 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-podinfo" pod "5022b50c-515a-4ca3-af29-ee8ce01de949" (UID: "5022b50c-515a-4ca3-af29-ee8ce01de949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:26.075391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.075355 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn"] Oct 30 01:01:26.076557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.076528 4517 cpu_manager.go:414] "containerMap: removing container" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerName="cleanup" Oct 30 01:01:26.076738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.076724 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerName="cleanup" Oct 30 01:01:26.076847 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.076837 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerName="cleanup" Oct 30 01:01:26.076920 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.076909 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerName="cleanup" Oct 30 01:01:26.076995 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.076987 4517 cpu_manager.go:414] "containerMap: removing container" podUID="c15581be-8d44-43be-832f-323c013ba2c2" containerName="cleanup" Oct 30 01:01:26.077066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077057 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15581be-8d44-43be-832f-323c013ba2c2" containerName="cleanup" Oct 30 01:01:26.077140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077131 4517 cpu_manager.go:414] "containerMap: removing container" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerName="cleanup" Oct 30 01:01:26.077218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077209 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerName="cleanup" Oct 30 01:01:26.077383 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077371 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" Oct 30 01:01:26.077471 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077462 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" Oct 30 01:01:26.077564 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077553 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerName="cleanup" Oct 30 01:01:26.077641 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077632 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerName="cleanup" Oct 30 01:01:26.077735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077724 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerName="cleanup" Oct 30 01:01:26.077818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077808 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerName="cleanup" Oct 30 01:01:26.077899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077891 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerName="cleanup" Oct 30 01:01:26.077973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.077964 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerName="cleanup" Oct 30 01:01:26.078049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078038 4517 cpu_manager.go:414] "containerMap: removing container" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerName="cleanup" Oct 30 01:01:26.078120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078111 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerName="cleanup" Oct 30 01:01:26.078197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078188 4517 cpu_manager.go:414] "containerMap: removing container" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerName="cleanup" Oct 30 01:01:26.078273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078263 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerName="cleanup" Oct 30 01:01:26.078348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078339 4517 cpu_manager.go:414] "containerMap: removing container" podUID="c15581be-8d44-43be-832f-323c013ba2c2" containerName="cleanup" Oct 30 01:01:26.078415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078406 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15581be-8d44-43be-832f-323c013ba2c2" containerName="cleanup" Oct 30 01:01:26.078492 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078483 4517 cpu_manager.go:414] "containerMap: removing container" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerName="cleanup" Oct 30 01:01:26.078563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078554 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerName="cleanup" Oct 30 01:01:26.078651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078642 4517 cpu_manager.go:414] "containerMap: removing container" podUID="36b8e0b5-d0bf-4a50-b448-de3117133944" containerName="main" Oct 30 01:01:26.078779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078767 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b8e0b5-d0bf-4a50-b448-de3117133944" containerName="main" Oct 30 01:01:26.078902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078892 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1687a186-3e99-47e3-8b70-d131495afade" containerName="cleanup" Oct 30 01:01:26.078976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.078967 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1687a186-3e99-47e3-8b70-d131495afade" containerName="cleanup" Oct 30 01:01:26.079081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079054 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1687a186-3e99-47e3-8b70-d131495afade" containerName="cleanup" Oct 30 01:01:26.079156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079146 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1687a186-3e99-47e3-8b70-d131495afade" containerName="cleanup" Oct 30 01:01:26.079245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079235 4517 cpu_manager.go:414] "containerMap: removing container" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerName="cleanup" Oct 30 01:01:26.079318 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079309 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerName="cleanup" Oct 30 01:01:26.079402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079393 4517 cpu_manager.go:414] "containerMap: removing container" podUID="c15581be-8d44-43be-832f-323c013ba2c2" containerName="cleanup" Oct 30 01:01:26.079476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079466 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15581be-8d44-43be-832f-323c013ba2c2" containerName="cleanup" Oct 30 01:01:26.079552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079543 4517 cpu_manager.go:414] "containerMap: removing container" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" Oct 30 01:01:26.079625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079615 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" Oct 30 01:01:26.079729 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079698 4517 cpu_manager.go:414] "containerMap: removing container" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerName="cleanup" Oct 30 01:01:26.079822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079811 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerName="cleanup" Oct 30 01:01:26.079902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079892 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerName="cleanup" Oct 30 01:01:26.079970 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.079960 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerName="cleanup" Oct 30 01:01:26.080051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080040 4517 cpu_manager.go:414] "containerMap: removing container" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerName="cleanup" Oct 30 01:01:26.080124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080115 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerName="cleanup" Oct 30 01:01:26.080219 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080210 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" containerName="cleanup" Oct 30 01:01:26.080289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080280 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" containerName="cleanup" Oct 30 01:01:26.080366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080357 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" containerName="cleanup" Oct 30 01:01:26.080439 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080430 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" containerName="cleanup" Oct 30 01:01:26.080511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080502 4517 cpu_manager.go:414] "containerMap: removing container" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerName="cleanup" Oct 30 01:01:26.080583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080574 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerName="cleanup" Oct 30 01:01:26.080665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080657 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1687a186-3e99-47e3-8b70-d131495afade" containerName="cleanup" Oct 30 01:01:26.080759 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080748 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1687a186-3e99-47e3-8b70-d131495afade" containerName="cleanup" Oct 30 01:01:26.080837 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080829 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" containerName="cleanup" Oct 30 01:01:26.080915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.080906 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" containerName="cleanup" Oct 30 01:01:26.081186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081174 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerName="cleanup" Oct 30 01:01:26.081279 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081270 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1687a186-3e99-47e3-8b70-d131495afade" containerName="cleanup" Oct 30 01:01:26.081354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081345 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1687a186-3e99-47e3-8b70-d131495afade" containerName="cleanup" Oct 30 01:01:26.081432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081424 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerName="cleanup" Oct 30 01:01:26.081503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081494 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerName="cleanup" Oct 30 01:01:26.081581 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081572 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15581be-8d44-43be-832f-323c013ba2c2" containerName="cleanup" Oct 30 01:01:26.081651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081643 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerName="istio-proxy" Oct 30 01:01:26.081749 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081739 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerName="cleanup" Oct 30 01:01:26.081821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081810 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerName="cleanup" Oct 30 01:01:26.081898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081889 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerName="cleanup" Oct 30 01:01:26.081974 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.081966 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerName="cleanup" Oct 30 01:01:26.082050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082041 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" containerName="cleanup" Oct 30 01:01:26.082129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082120 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" containerName="cleanup" Oct 30 01:01:26.082200 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082191 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="113b92e9-3769-4c4d-9a1d-e4f81277c68f" containerName="cleanup" Oct 30 01:01:26.082277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082267 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5861cf21-9842-4a34-884d-85ce63b421ee" containerName="cleanup" Oct 30 01:01:26.082357 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082348 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15581be-8d44-43be-832f-323c013ba2c2" containerName="cleanup" Oct 30 01:01:26.082427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082420 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerName="cleanup" Oct 30 01:01:26.082498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082490 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerName="istio-proxy" Oct 30 01:01:26.082583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082574 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerName="cleanup" Oct 30 01:01:26.082660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082651 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerName="cleanup" Oct 30 01:01:26.082755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082744 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ba61a-e155-4f51-9e2d-8ba7ca6c3972" containerName="cleanup" Oct 30 01:01:26.082838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082828 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerName="cleanup" Oct 30 01:01:26.082912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082904 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerName="cleanup" Oct 30 01:01:26.082979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.082970 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15581be-8d44-43be-832f-323c013ba2c2" containerName="cleanup" Oct 30 01:01:26.083089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.083079 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerName="cleanup" Oct 30 01:01:26.083167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.083158 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b8e0b5-d0bf-4a50-b448-de3117133944" containerName="main" Oct 30 01:01:26.088801 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.088776 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:26.110830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.110022 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-49dfaf\"" Oct 30 01:01:26.110830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.110778 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-49dfaf-dockercfg-j9hph\"" Oct 30 01:01:26.112635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.112538 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn"] Oct 30 01:01:26.123410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123378 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkwft\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-kube-api-access-fkwft\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.123410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123413 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.123643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123429 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.123643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123442 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ftn\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-kube-api-access-l2ftn\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.123643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123456 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.123643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123467 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.123643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123478 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.123643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123490 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5022b50c-515a-4ca3-af29-ee8ce01de949-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.123643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123502 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.123643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.123514 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c81e6f4-4b34-4711-ba3f-09be57119c0a-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.169145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.169113 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 01:01:26.171351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.171319 4517 generic.go:356] "Generic (PLEG): container finished" podID="5022b50c-515a-4ca3-af29-ee8ce01de949" containerID="7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b" exitCode=0 Oct 30 01:01:26.171519 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.171371 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" Oct 30 01:01:26.171519 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.171407 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" event={"ID":"5022b50c-515a-4ca3-af29-ee8ce01de949","Type":"ContainerDied","Data":"7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b"} Oct 30 01:01:26.171519 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.171472 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt" event={"ID":"5022b50c-515a-4ca3-af29-ee8ce01de949","Type":"ContainerDied","Data":"d419ab6949d394d3c708b5ec4c5360c8b79d36c8512bac1ef5afc4ead00c8a17"} Oct 30 01:01:26.171519 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.171504 4517 scope.go:122] "RemoveContainer" containerID="7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b" Oct 30 01:01:26.174210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.174181 4517 generic.go:356] "Generic (PLEG): container finished" podID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" containerID="a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b" exitCode=0 Oct 30 01:01:26.174309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.174275 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" Oct 30 01:01:26.176295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.176267 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" event={"ID":"2c81e6f4-4b34-4711-ba3f-09be57119c0a","Type":"ContainerDied","Data":"a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b"} Oct 30 01:01:26.176405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.176328 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m" event={"ID":"2c81e6f4-4b34-4711-ba3f-09be57119c0a","Type":"ContainerDied","Data":"3ddd6f5d4cd28b8886e1b6e5395c4e90b05f7c68114a194562925bee770df303"} Oct 30 01:01:26.192140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.190493 4517 scope.go:122] "RemoveContainer" containerID="7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b" Oct 30 01:01:26.192140 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:01:26.191305 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b\": container with ID starting with 7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b not found: ID does not exist" containerID="7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b" Oct 30 01:01:26.192140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.191346 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b"} err="failed to get container status \"7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b\": rpc error: code = NotFound desc = could not find container \"7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b\": container with ID starting with 7c2ae5c667bb0222bb1d527d2556fd8273ad3d4daa14e62312e748174c42792b not found: ID does not exist" Oct 30 01:01:26.192140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.191372 4517 scope.go:122] "RemoveContainer" containerID="a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b" Oct 30 01:01:26.199455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.197965 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:26.205727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.205661 4517 generic.go:356] "Generic (PLEG): container finished" podID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerID="adcee1d7d371cc3f3d8183e9463d484463d3ca36ff67ea5f229d0e4c1679c447" exitCode=0 Oct 30 01:01:26.205892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.205837 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" event={"ID":"0b387e5c-219f-4249-a3ee-4903af9a493f","Type":"ContainerDied","Data":"adcee1d7d371cc3f3d8183e9463d484463d3ca36ff67ea5f229d0e4c1679c447"} Oct 30 01:01:26.205962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.205943 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22" Oct 30 01:01:26.220782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.219153 4517 scope.go:122] "RemoveContainer" containerID="a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b" Oct 30 01:01:26.220782 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:01:26.220039 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b\": container with ID starting with a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b not found: ID does not exist" containerID="a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b" Oct 30 01:01:26.220782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.220089 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b"} err="failed to get container status \"a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b\": rpc error: code = NotFound desc = could not find container \"a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b\": container with ID starting with a5e99f1f1c4d451f0f32ee0ceafe880590d22c203e3f5f024cc508d01b98ce7b not found: ID does not exist" Oct 30 01:01:26.220782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.220117 4517 scope.go:122] "RemoveContainer" containerID="adcee1d7d371cc3f3d8183e9463d484463d3ca36ff67ea5f229d0e4c1679c447" Oct 30 01:01:26.245626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.233060 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn\" (UID: \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:26.245626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.233385 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqfb\" (UniqueName: \"kubernetes.io/projected/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-kube-api-access-bnqfb\") pod \"payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn\" (UID: \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:26.311058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.311031 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt"] Oct 30 01:01:26.317739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.317120 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jtdpt"] Oct 30 01:01:26.334577 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.334549 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-credential-socket\") pod \"0b387e5c-219f-4249-a3ee-4903af9a493f\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " Oct 30 01:01:26.334688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.334604 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-data\") pod \"0b387e5c-219f-4249-a3ee-4903af9a493f\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " Oct 30 01:01:26.334688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.334663 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-podinfo\") pod \"0b387e5c-219f-4249-a3ee-4903af9a493f\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " Oct 30 01:01:26.334834 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.334771 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-envoy\") pod \"0b387e5c-219f-4249-a3ee-4903af9a493f\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " Oct 30 01:01:26.334948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.334829 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-kube-api-access-xb44q\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-kube-api-access-xb44q\") pod \"0b387e5c-219f-4249-a3ee-4903af9a493f\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " Oct 30 01:01:26.334948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.334916 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-certs\") pod \"0b387e5c-219f-4249-a3ee-4903af9a493f\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " Oct 30 01:01:26.335097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.334979 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-credential-socket" pod "0b387e5c-219f-4249-a3ee-4903af9a493f" (UID: "0b387e5c-219f-4249-a3ee-4903af9a493f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.335097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.335013 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b387e5c-219f-4249-a3ee-4903af9a493f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b387e5c-219f-4249-a3ee-4903af9a493f-istiod-ca-cert\") pod \"0b387e5c-219f-4249-a3ee-4903af9a493f\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " Oct 30 01:01:26.335281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.335109 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-token\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-token\") pod \"0b387e5c-219f-4249-a3ee-4903af9a493f\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " Oct 30 01:01:26.335281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.335195 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-socket\") pod \"0b387e5c-219f-4249-a3ee-4903af9a493f\" (UID: \"0b387e5c-219f-4249-a3ee-4903af9a493f\") " Oct 30 01:01:26.335400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.335302 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-data" pod "0b387e5c-219f-4249-a3ee-4903af9a493f" (UID: "0b387e5c-219f-4249-a3ee-4903af9a493f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.335462 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.335413 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b387e5c-219f-4249-a3ee-4903af9a493f-istiod-ca-cert" pod "0b387e5c-219f-4249-a3ee-4903af9a493f" (UID: "0b387e5c-219f-4249-a3ee-4903af9a493f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:26.335523 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.335476 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-certs" pod "0b387e5c-219f-4249-a3ee-4903af9a493f" (UID: "0b387e5c-219f-4249-a3ee-4903af9a493f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.335677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.335652 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqfb\" (UniqueName: \"kubernetes.io/projected/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-kube-api-access-bnqfb\") pod \"payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn\" (UID: \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:26.335968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.335657 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-socket" pod "0b387e5c-219f-4249-a3ee-4903af9a493f" (UID: "0b387e5c-219f-4249-a3ee-4903af9a493f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.336063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.336037 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn\" (UID: \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:26.336827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.336219 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.338477 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.337685 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn\" (UID: \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:26.338477 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.337904 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b387e5c-219f-4249-a3ee-4903af9a493f-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.338670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.338648 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-podinfo" pod "0b387e5c-219f-4249-a3ee-4903af9a493f" (UID: "0b387e5c-219f-4249-a3ee-4903af9a493f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:26.338670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.338658 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-token" pod "0b387e5c-219f-4249-a3ee-4903af9a493f" (UID: "0b387e5c-219f-4249-a3ee-4903af9a493f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.339172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.338986 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-kube-api-access-xb44q" pod "0b387e5c-219f-4249-a3ee-4903af9a493f" (UID: "0b387e5c-219f-4249-a3ee-4903af9a493f"). InnerVolumeSpecName "kube-api-access-xb44q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.339464 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.339420 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.339541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.339469 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.339541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.339485 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.339946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.339924 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-envoy" pod "0b387e5c-219f-4249-a3ee-4903af9a493f" (UID: "0b387e5c-219f-4249-a3ee-4903af9a493f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.342607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.342581 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m"] Oct 30 01:01:26.348692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.348659 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-dfz2m"] Oct 30 01:01:26.373622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.373584 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:26.391398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.391371 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqfb\" (UniqueName: \"kubernetes.io/projected/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-kube-api-access-bnqfb\") pod \"payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn\" (UID: \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:26.428616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.428344 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:26.449846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.442481 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.449846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.442519 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.449846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.442533 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb44q\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-kube-api-access-xb44q\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.449846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.442546 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b387e5c-219f-4249-a3ee-4903af9a493f-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.656797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.651922 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22"] Oct 30 01:01:26.693808 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.671099 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-hph22"] Oct 30 01:01:26.731355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.731245 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:26.993098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:26.993004 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn"] Oct 30 01:01:27.033823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.032328 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:27.204983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.204954 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 01:01:27.215132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.214960 4517 generic.go:356] "Generic (PLEG): container finished" podID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerID="32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9" exitCode=0 Oct 30 01:01:27.215132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.215040 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" Oct 30 01:01:27.215132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.215087 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" event={"ID":"9d6f3ac9-1b47-4fe0-a82e-165a947bd458","Type":"ContainerDied","Data":"32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9"} Oct 30 01:01:27.215418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.215208 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd" event={"ID":"9d6f3ac9-1b47-4fe0-a82e-165a947bd458","Type":"ContainerDied","Data":"1bf761b54700146cedb307790d134565e84384cba661613f2d0b93eb45901d4c"} Oct 30 01:01:27.215418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.215232 4517 scope.go:122] "RemoveContainer" containerID="32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9" Oct 30 01:01:27.224122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.224090 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" event={"ID":"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd","Type":"ContainerStarted","Data":"4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729"} Oct 30 01:01:27.224285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.224189 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" event={"ID":"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd","Type":"ContainerStarted","Data":"6b11e66d5820d354a01df55d53302fe6640cd646a7a66f7542bc17beb0f00c3a"} Oct 30 01:01:27.224486 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.224469 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:27.230396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.230365 4517 scope.go:122] "RemoveContainer" containerID="32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9" Oct 30 01:01:27.230758 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:01:27.230737 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9\": container with ID starting with 32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9 not found: ID does not exist" containerID="32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9" Oct 30 01:01:27.230849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.230765 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9"} err="failed to get container status \"32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9\": rpc error: code = NotFound desc = could not find container \"32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9\": container with ID starting with 32f0cd1e95e3bd4b80707e68a105e1ba122d36597da00532e95d9e6d3b1c49f9 not found: ID does not exist" Oct 30 01:01:27.283489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.283449 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-credential-socket\") pod \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " Oct 30 01:01:27.283690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.283620 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-kube-api-access-2tl5d\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-kube-api-access-2tl5d\") pod \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " Oct 30 01:01:27.283836 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.283787 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-socket\") pod \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " Oct 30 01:01:27.283898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.283836 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-certs\") pod \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " Oct 30 01:01:27.283975 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.283944 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-data\") pod \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " Oct 30 01:01:27.284029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.284002 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istiod-ca-cert\") pod \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " Oct 30 01:01:27.284082 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.284034 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-token\") pod \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " Oct 30 01:01:27.284139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.284114 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-podinfo\") pod \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " Oct 30 01:01:27.284211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.284181 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-envoy\") pod \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\" (UID: \"9d6f3ac9-1b47-4fe0-a82e-165a947bd458\") " Oct 30 01:01:27.285220 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.285193 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-data" pod "9d6f3ac9-1b47-4fe0-a82e-165a947bd458" (UID: "9d6f3ac9-1b47-4fe0-a82e-165a947bd458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:27.285603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.285567 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-credential-socket" pod "9d6f3ac9-1b47-4fe0-a82e-165a947bd458" (UID: "9d6f3ac9-1b47-4fe0-a82e-165a947bd458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:27.286647 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.286449 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-socket" pod "9d6f3ac9-1b47-4fe0-a82e-165a947bd458" (UID: "9d6f3ac9-1b47-4fe0-a82e-165a947bd458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:27.287041 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.286898 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istiod-ca-cert" pod "9d6f3ac9-1b47-4fe0-a82e-165a947bd458" (UID: "9d6f3ac9-1b47-4fe0-a82e-165a947bd458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:27.287148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.286972 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-certs" pod "9d6f3ac9-1b47-4fe0-a82e-165a947bd458" (UID: "9d6f3ac9-1b47-4fe0-a82e-165a947bd458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:27.292333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.291816 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-envoy" pod "9d6f3ac9-1b47-4fe0-a82e-165a947bd458" (UID: "9d6f3ac9-1b47-4fe0-a82e-165a947bd458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:27.295008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.294979 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-podinfo" pod "9d6f3ac9-1b47-4fe0-a82e-165a947bd458" (UID: "9d6f3ac9-1b47-4fe0-a82e-165a947bd458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:27.297206 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.297168 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-token" pod "9d6f3ac9-1b47-4fe0-a82e-165a947bd458" (UID: "9d6f3ac9-1b47-4fe0-a82e-165a947bd458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:27.297206 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.297181 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-kube-api-access-2tl5d" pod "9d6f3ac9-1b47-4fe0-a82e-165a947bd458" (UID: "9d6f3ac9-1b47-4fe0-a82e-165a947bd458"). InnerVolumeSpecName "kube-api-access-2tl5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:27.343888 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.343828 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" podStartSLOduration=1.343814842 podStartE2EDuration="1.343814842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:27.340488545 +0000 UTC m=+1625.252615685" watchObservedRunningTime="2026-10-30 01:01:27.343814842 +0000 UTC m=+1625.255941982" Oct 30 01:01:27.386344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.386259 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:27.386344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.386290 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:27.386344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.386299 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:27.386344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.386307 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:27.386344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.386317 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:27.386344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.386327 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tl5d\" (UniqueName: \"kubernetes.io/projected/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-kube-api-access-2tl5d\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:27.386344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.386335 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:27.386344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.386342 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:27.386737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.386355 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6f3ac9-1b47-4fe0-a82e-165a947bd458-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:27.539773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.539744 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:27.570759 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.570723 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd"] Oct 30 01:01:27.579192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:27.579161 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lxbkd"] Oct 30 01:01:28.110428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:28.110391 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:28.262794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:28.262755 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" path="/var/lib/kubelet/pods/0b387e5c-219f-4249-a3ee-4903af9a493f/volumes" Oct 30 01:01:28.263866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:28.263823 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c81e6f4-4b34-4711-ba3f-09be57119c0a" path="/var/lib/kubelet/pods/2c81e6f4-4b34-4711-ba3f-09be57119c0a/volumes" Oct 30 01:01:28.264745 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:28.264725 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5022b50c-515a-4ca3-af29-ee8ce01de949" path="/var/lib/kubelet/pods/5022b50c-515a-4ca3-af29-ee8ce01de949/volumes" Oct 30 01:01:28.271443 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:28.270301 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" path="/var/lib/kubelet/pods/9d6f3ac9-1b47-4fe0-a82e-165a947bd458/volumes" Oct 30 01:01:28.817347 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:28.817309 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:29.554801 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:29.554764 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:30.758602 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:01:30.758558 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:01:31.467271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:31.467073 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:32.056602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:32.056564 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:32.205366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:32.205337 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:33.622941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:33.622905 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:33.747278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:33.747246 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:33.877443 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:33.877326 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:34.524068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.524033 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:34.671993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.671958 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.965762 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr"] Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.976829 4517 cpu_manager.go:414] "containerMap: removing container" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerName="cleanup" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.976875 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb7aed0-c8eb-4969-a2c6-b1ed09b4e1a5" containerName="cleanup" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.976906 4517 cpu_manager.go:414] "containerMap: removing container" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerName="cleanup" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.977123 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerName="cleanup" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.977145 4517 cpu_manager.go:414] "containerMap: removing container" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerName="cleanup" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.977154 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerName="cleanup" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.977180 4517 cpu_manager.go:414] "containerMap: removing container" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.977188 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.977457 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerName="cleanup" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.977467 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerName="cleanup" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.977699 4517 cpu_manager.go:414] "containerMap: removing container" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerName="istio-proxy" Oct 30 01:01:34.979746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:34.977729 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerName="istio-proxy" Oct 30 01:01:35.061952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.061923 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5696d68c-2f3d-4a6a-af56-7228baf535af" containerName="cleanup" Oct 30 01:01:35.062150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.061965 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b387e5c-219f-4249-a3ee-4903af9a493f" containerName="istio-proxy" Oct 30 01:01:35.062150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.061990 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3e6997-7ad8-4065-96ea-78928efa2227" containerName="cleanup" Oct 30 01:01:35.062150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.062003 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eedeef4-62fc-48d9-8eca-89f79ac5e4b0" containerName="cleanup" Oct 30 01:01:35.062150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.062012 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6f3ac9-1b47-4fe0-a82e-165a947bd458" containerName="istio-proxy" Oct 30 01:01:35.062150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.062031 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1687a186-3e99-47e3-8b70-d131495afade" containerName="cleanup" Oct 30 01:01:35.088920 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.088883 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr"] Oct 30 01:01:35.089159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.089005 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:35.102566 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.102541 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-cf3466\"" Oct 30 01:01:35.112880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.112848 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-cf3466-dockercfg-68csh\"" Oct 30 01:01:35.277796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.264481 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7pt\" (UniqueName: \"kubernetes.io/projected/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-kube-api-access-jh7pt\") pod \"payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr\" (UID: \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:35.277796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.264585 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr\" (UID: \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:35.366573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.366537 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7pt\" (UniqueName: \"kubernetes.io/projected/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-kube-api-access-jh7pt\") pod \"payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr\" (UID: \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:35.366846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.366654 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr\" (UID: \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:35.367639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.367615 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr\" (UID: \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:35.389435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.389389 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7pt\" (UniqueName: \"kubernetes.io/projected/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-kube-api-access-jh7pt\") pod \"payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr\" (UID: \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:35.405303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.405254 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:35.571532 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.571501 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw"] Oct 30 01:01:35.571743 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.571689 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" containerID="cri-o://d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84" gracePeriod=30 Oct 30 01:01:35.724079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.720621 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr"] Oct 30 01:01:35.729074 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:01:35.729025 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d693d57_a4e9_4efa_a6eb_0a19cbd6a547.slice/crio-31a6d490d46f7bc77608574768eb8e15f66a3ccb606394ac6e7ed43aa8c69b6e WatchSource:0}: Error finding container 31a6d490d46f7bc77608574768eb8e15f66a3ccb606394ac6e7ed43aa8c69b6e: Status 404 returned error can't find the container with id 31a6d490d46f7bc77608574768eb8e15f66a3ccb606394ac6e7ed43aa8c69b6e Oct 30 01:01:35.771726 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.771678 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7"] Oct 30 01:01:35.772005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.771956 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" containerID="cri-o://ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21" gracePeriod=30 Oct 30 01:01:35.828790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:35.826136 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:36.309768 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:36.309702 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" event={"ID":"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547","Type":"ContainerStarted","Data":"48547028e1480cbc8b496cde7522f32e6f1513cf295077839726f9516c0fb6aa"} Oct 30 01:01:36.309987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:36.309790 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" event={"ID":"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547","Type":"ContainerStarted","Data":"31a6d490d46f7bc77608574768eb8e15f66a3ccb606394ac6e7ed43aa8c69b6e"} Oct 30 01:01:36.309987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:36.309945 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:36.352448 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:36.352386 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" podStartSLOduration=2.3523735820000002 podStartE2EDuration="2.352373582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:36.347137544 +0000 UTC m=+1634.259264684" watchObservedRunningTime="2026-10-30 01:01:36.352373582 +0000 UTC m=+1634.264500722" Oct 30 01:01:37.544803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:37.544765 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:38.250170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:38.250145 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:01:39.312615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:39.312570 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:41.047842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.047505 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 01:01:41.115422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.115335 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-data\") pod \"325b3ba8-8192-4bf6-8a03-846749be71c5\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " Oct 30 01:01:41.115865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.115802 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-data" pod "325b3ba8-8192-4bf6-8a03-846749be71c5" (UID: "325b3ba8-8192-4bf6-8a03-846749be71c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.116124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.116095 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325b3ba8-8192-4bf6-8a03-846749be71c5-istiod-ca-cert" pod "325b3ba8-8192-4bf6-8a03-846749be71c5" (UID: "325b3ba8-8192-4bf6-8a03-846749be71c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:41.116242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.116150 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325b3ba8-8192-4bf6-8a03-846749be71c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b3ba8-8192-4bf6-8a03-846749be71c5-istiod-ca-cert\") pod \"325b3ba8-8192-4bf6-8a03-846749be71c5\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " Oct 30 01:01:41.116242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.116216 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-podinfo\") pod \"325b3ba8-8192-4bf6-8a03-846749be71c5\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " Oct 30 01:01:41.116616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.116595 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-certs\") pod \"325b3ba8-8192-4bf6-8a03-846749be71c5\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " Oct 30 01:01:41.116971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.116943 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-certs" pod "325b3ba8-8192-4bf6-8a03-846749be71c5" (UID: "325b3ba8-8192-4bf6-8a03-846749be71c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.117078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.116990 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-token\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-token\") pod \"325b3ba8-8192-4bf6-8a03-846749be71c5\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " Oct 30 01:01:41.117078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.117047 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-kube-api-access-dqcd4\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-kube-api-access-dqcd4\") pod \"325b3ba8-8192-4bf6-8a03-846749be71c5\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " Oct 30 01:01:41.117202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.117122 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-envoy\") pod \"325b3ba8-8192-4bf6-8a03-846749be71c5\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " Oct 30 01:01:41.117202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.117169 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-socket\") pod \"325b3ba8-8192-4bf6-8a03-846749be71c5\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " Oct 30 01:01:41.117583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.117555 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-socket" pod "325b3ba8-8192-4bf6-8a03-846749be71c5" (UID: "325b3ba8-8192-4bf6-8a03-846749be71c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.117737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.117684 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-credential-socket\") pod \"325b3ba8-8192-4bf6-8a03-846749be71c5\" (UID: \"325b3ba8-8192-4bf6-8a03-846749be71c5\") " Oct 30 01:01:41.117921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.117899 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-credential-socket" pod "325b3ba8-8192-4bf6-8a03-846749be71c5" (UID: "325b3ba8-8192-4bf6-8a03-846749be71c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.118493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.118468 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.118606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.118499 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.118606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.118514 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b3ba8-8192-4bf6-8a03-846749be71c5-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.118606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.118527 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.118606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.118539 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.120347 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.120322 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-envoy" pod "325b3ba8-8192-4bf6-8a03-846749be71c5" (UID: "325b3ba8-8192-4bf6-8a03-846749be71c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.121822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.121796 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-token" pod "325b3ba8-8192-4bf6-8a03-846749be71c5" (UID: "325b3ba8-8192-4bf6-8a03-846749be71c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:41.122589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.122559 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-kube-api-access-dqcd4" pod "325b3ba8-8192-4bf6-8a03-846749be71c5" (UID: "325b3ba8-8192-4bf6-8a03-846749be71c5"). InnerVolumeSpecName "kube-api-access-dqcd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:41.123864 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.123836 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-podinfo" pod "325b3ba8-8192-4bf6-8a03-846749be71c5" (UID: "325b3ba8-8192-4bf6-8a03-846749be71c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:41.174087 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.174061 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 01:01:41.224519 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.224469 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-envoy\") pod \"1c8fd775-d621-44cd-84b2-9030019d7f44\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " Oct 30 01:01:41.224798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.224612 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8fd775-d621-44cd-84b2-9030019d7f44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8fd775-d621-44cd-84b2-9030019d7f44-istiod-ca-cert\") pod \"1c8fd775-d621-44cd-84b2-9030019d7f44\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " Oct 30 01:01:41.224798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.224759 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-data\") pod \"1c8fd775-d621-44cd-84b2-9030019d7f44\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " Oct 30 01:01:41.224909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.224812 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-token\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-token\") pod \"1c8fd775-d621-44cd-84b2-9030019d7f44\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " Oct 30 01:01:41.224948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.224914 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-kube-api-access-r9lgz\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-kube-api-access-r9lgz\") pod \"1c8fd775-d621-44cd-84b2-9030019d7f44\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " Oct 30 01:01:41.225076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.225059 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-certs\") pod \"1c8fd775-d621-44cd-84b2-9030019d7f44\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " Oct 30 01:01:41.225142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.225097 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-podinfo\") pod \"1c8fd775-d621-44cd-84b2-9030019d7f44\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " Oct 30 01:01:41.225185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.225141 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-socket\") pod \"1c8fd775-d621-44cd-84b2-9030019d7f44\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " Oct 30 01:01:41.225219 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.225189 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-credential-socket\") pod \"1c8fd775-d621-44cd-84b2-9030019d7f44\" (UID: \"1c8fd775-d621-44cd-84b2-9030019d7f44\") " Oct 30 01:01:41.225904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.225882 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqcd4\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-kube-api-access-dqcd4\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.225904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.225905 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.226080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.225914 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.226080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.225929 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b3ba8-8192-4bf6-8a03-846749be71c5-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.226802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.226655 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-certs" pod "1c8fd775-d621-44cd-84b2-9030019d7f44" (UID: "1c8fd775-d621-44cd-84b2-9030019d7f44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.226802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.226774 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-credential-socket" pod "1c8fd775-d621-44cd-84b2-9030019d7f44" (UID: "1c8fd775-d621-44cd-84b2-9030019d7f44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.226956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.226814 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-socket" pod "1c8fd775-d621-44cd-84b2-9030019d7f44" (UID: "1c8fd775-d621-44cd-84b2-9030019d7f44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.227007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.226986 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-data" pod "1c8fd775-d621-44cd-84b2-9030019d7f44" (UID: "1c8fd775-d621-44cd-84b2-9030019d7f44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.227236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.227212 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8fd775-d621-44cd-84b2-9030019d7f44-istiod-ca-cert" pod "1c8fd775-d621-44cd-84b2-9030019d7f44" (UID: "1c8fd775-d621-44cd-84b2-9030019d7f44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:41.229937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.229901 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-envoy" pod "1c8fd775-d621-44cd-84b2-9030019d7f44" (UID: "1c8fd775-d621-44cd-84b2-9030019d7f44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.230179 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.230123 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-podinfo" pod "1c8fd775-d621-44cd-84b2-9030019d7f44" (UID: "1c8fd775-d621-44cd-84b2-9030019d7f44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:41.230179 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.230152 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-token" pod "1c8fd775-d621-44cd-84b2-9030019d7f44" (UID: "1c8fd775-d621-44cd-84b2-9030019d7f44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:41.232257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.232230 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-kube-api-access-r9lgz" pod "1c8fd775-d621-44cd-84b2-9030019d7f44" (UID: "1c8fd775-d621-44cd-84b2-9030019d7f44"). InnerVolumeSpecName "kube-api-access-r9lgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:41.326952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.326924 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9lgz\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-kube-api-access-r9lgz\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.326952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.326950 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.326952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.326961 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.327228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.326969 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.327228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.326977 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.327228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.326985 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.327228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.326994 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8fd775-d621-44cd-84b2-9030019d7f44-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.327228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.327001 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.327228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.327008 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8fd775-d621-44cd-84b2-9030019d7f44-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.351158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.351130 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:41.353257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.353232 4517 generic.go:356] "Generic (PLEG): container finished" podID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerID="ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21" exitCode=0 Oct 30 01:01:41.353429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.353277 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" Oct 30 01:01:41.353429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.353322 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" event={"ID":"1c8fd775-d621-44cd-84b2-9030019d7f44","Type":"ContainerDied","Data":"ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21"} Oct 30 01:01:41.353429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.353374 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7" event={"ID":"1c8fd775-d621-44cd-84b2-9030019d7f44","Type":"ContainerDied","Data":"e5285bf1c930872d725472d9301e9661425ff4d5c5cb9f5c2f0407d6de05c39c"} Oct 30 01:01:41.353429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.353400 4517 scope.go:122] "RemoveContainer" containerID="ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21" Oct 30 01:01:41.355086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.355062 4517 generic.go:356] "Generic (PLEG): container finished" podID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerID="d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84" exitCode=0 Oct 30 01:01:41.355187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.355103 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" Oct 30 01:01:41.355187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.355140 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" event={"ID":"325b3ba8-8192-4bf6-8a03-846749be71c5","Type":"ContainerDied","Data":"d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84"} Oct 30 01:01:41.355280 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.355202 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw" event={"ID":"325b3ba8-8192-4bf6-8a03-846749be71c5","Type":"ContainerDied","Data":"3b560638c2377506f13c1c6d94318c639355b0e6557599d3f8094b9e72d4cfb4"} Oct 30 01:01:41.366397 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.366381 4517 scope.go:122] "RemoveContainer" containerID="ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21" Oct 30 01:01:41.366790 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:01:41.366767 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21\": container with ID starting with ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21 not found: ID does not exist" containerID="ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21" Oct 30 01:01:41.366921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.366892 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21"} err="failed to get container status \"ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21\": rpc error: code = NotFound desc = could not find container \"ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21\": container with ID starting with ea2dbbcaaefe57d95eb8e02444dc8853c674078c47207fe003fc5e9547c02f21 not found: ID does not exist" Oct 30 01:01:41.366921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.366918 4517 scope.go:122] "RemoveContainer" containerID="d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84" Oct 30 01:01:41.394530 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.394505 4517 scope.go:122] "RemoveContainer" containerID="d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84" Oct 30 01:01:41.394967 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:01:41.394937 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84\": container with ID starting with d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84 not found: ID does not exist" containerID="d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84" Oct 30 01:01:41.395224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.395188 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84"} err="failed to get container status \"d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84\": rpc error: code = NotFound desc = could not find container \"d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84\": container with ID starting with d5beee63212bf8f67dee75184b49a46a4c6b6b25d698a391c07ce7cbb8612e84 not found: ID does not exist" Oct 30 01:01:41.416669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.416640 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7"] Oct 30 01:01:41.425327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.425279 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-lbpf7"] Oct 30 01:01:41.468757 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.465811 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw"] Oct 30 01:01:41.470042 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:41.470016 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-r7ggw"] Oct 30 01:01:42.235481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:42.235444 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" path="/var/lib/kubelet/pods/1c8fd775-d621-44cd-84b2-9030019d7f44/volumes" Oct 30 01:01:42.235929 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:42.235914 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" path="/var/lib/kubelet/pods/325b3ba8-8192-4bf6-8a03-846749be71c5/volumes" Oct 30 01:01:43.166037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:43.165988 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:44.936452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:44.936407 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:45.054322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:45.054282 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:45.429773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:45.429742 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:45.948791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:45.948761 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:46.480345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:46.480303 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:47.319729 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:47.319674 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:01:47.346683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:47.346650 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:48.988568 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:48.988532 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk"] Oct 30 01:01:48.989349 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:48.989327 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" Oct 30 01:01:48.989452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:48.989351 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" Oct 30 01:01:48.989452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:48.989368 4517 cpu_manager.go:414] "containerMap: removing container" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" Oct 30 01:01:48.989452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:48.989375 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" Oct 30 01:01:48.989606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:48.989536 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8fd775-d621-44cd-84b2-9030019d7f44" containerName="istio-proxy" Oct 30 01:01:48.989606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:48.989544 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="325b3ba8-8192-4bf6-8a03-846749be71c5" containerName="istio-proxy" Oct 30 01:01:48.993278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:48.993256 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.010879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.010854 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-969b92-serving-cert\"" Oct 30 01:01:49.024688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.024654 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk"] Oct 30 01:01:49.102777 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.102698 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:49.155103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.155071 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.155285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.155205 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.155285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.155239 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.155285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.155264 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5g6\" (UniqueName: \"kubernetes.io/projected/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-kube-api-access-6p5g6\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.254813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.254256 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:49.256829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.256774 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.259787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.256943 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.259930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.259811 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.259930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.259857 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5g6\" (UniqueName: \"kubernetes.io/projected/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-kube-api-access-6p5g6\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.282739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.264281 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.282739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.267745 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.282739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.280749 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.309093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.308869 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5g6\" (UniqueName: \"kubernetes.io/projected/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-kube-api-access-6p5g6\") pod \"maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.330495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.330451 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:49.655512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.655483 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk"] Oct 30 01:01:49.657853 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:01:49.657827 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784a87ff_53c7_4543_aac8_4c47e1b0e1f3.slice/crio-9c3cc499ed9d89d587d6665fee261b6fa572c733939e079ce9225a8c75472b85 WatchSource:0}: Error finding container 9c3cc499ed9d89d587d6665fee261b6fa572c733939e079ce9225a8c75472b85: Status 404 returned error can't find the container with id 9c3cc499ed9d89d587d6665fee261b6fa572c733939e079ce9225a8c75472b85 Oct 30 01:01:49.659640 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:49.659616 4517 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:01:50.295718 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:50.295673 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:50.448925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:50.448891 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" event={"ID":"784a87ff-53c7-4543-aac8-4c47e1b0e1f3","Type":"ContainerStarted","Data":"27478e6f03c15d30e7af9f8ebd200fa3c1dbbf5e99ea744a8e5d756335b74fb6"} Oct 30 01:01:50.449073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:50.448953 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" event={"ID":"784a87ff-53c7-4543-aac8-4c47e1b0e1f3","Type":"ContainerStarted","Data":"9c3cc499ed9d89d587d6665fee261b6fa572c733939e079ce9225a8c75472b85"} Oct 30 01:01:50.449073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:50.448988 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:50.519370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:50.519314 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" podStartSLOduration=2.185573923 podStartE2EDuration="2.519299114s" totalImagesPullingTime="333.725191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:50.516177947 +0000 UTC m=+1648.428305088" watchObservedRunningTime="2026-10-30 01:01:50.519299114 +0000 UTC m=+1648.431426255" Oct 30 01:01:50.523957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:50.523930 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:51.679702 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:51.679661 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:52.533361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.533304 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:52.641874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.641846 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt"] Oct 30 01:01:52.645611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.645588 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:01:52.671835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.671806 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1da7b2\"" Oct 30 01:01:52.671835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.671806 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1da7b2-dockercfg-n9qh7\"" Oct 30 01:01:52.676883 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.676854 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt"] Oct 30 01:01:52.678121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.678088 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:52.766632 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.766582 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1503f898-c417-4ef8-8a3f-a33eed038eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt\" (UID: \"1503f898-c417-4ef8-8a3f-a33eed038eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:01:52.767206 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.766812 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2w64\" (UniqueName: \"kubernetes.io/projected/1503f898-c417-4ef8-8a3f-a33eed038eb6-kube-api-access-t2w64\") pod \"payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt\" (UID: \"1503f898-c417-4ef8-8a3f-a33eed038eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:01:52.877733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.868170 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1503f898-c417-4ef8-8a3f-a33eed038eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt\" (UID: \"1503f898-c417-4ef8-8a3f-a33eed038eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:01:52.877733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.868378 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2w64\" (UniqueName: \"kubernetes.io/projected/1503f898-c417-4ef8-8a3f-a33eed038eb6-kube-api-access-t2w64\") pod \"payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt\" (UID: \"1503f898-c417-4ef8-8a3f-a33eed038eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:01:52.877733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.869670 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1503f898-c417-4ef8-8a3f-a33eed038eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt\" (UID: \"1503f898-c417-4ef8-8a3f-a33eed038eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:01:52.976756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:52.974269 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2w64\" (UniqueName: \"kubernetes.io/projected/1503f898-c417-4ef8-8a3f-a33eed038eb6-kube-api-access-t2w64\") pod \"payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt\" (UID: \"1503f898-c417-4ef8-8a3f-a33eed038eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:01:53.260640 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:53.260545 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:01:53.651550 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:53.651524 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt"] Oct 30 01:01:53.653804 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:01:53.653778 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1503f898_c417_4ef8_8a3f_a33eed038eb6.slice/crio-cf56e48eb63f36fd03b09c4f034ecf80f4a2fb42eefd00ffcdcb38588dbc7d25 WatchSource:0}: Error finding container cf56e48eb63f36fd03b09c4f034ecf80f4a2fb42eefd00ffcdcb38588dbc7d25: Status 404 returned error can't find the container with id cf56e48eb63f36fd03b09c4f034ecf80f4a2fb42eefd00ffcdcb38588dbc7d25 Oct 30 01:01:54.573873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:54.573833 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" event={"ID":"1503f898-c417-4ef8-8a3f-a33eed038eb6","Type":"ContainerStarted","Data":"62b02136c4875fa73693495942e6371bb33c0b105adabede63e103c52ea45663"} Oct 30 01:01:54.574386 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:54.573896 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" event={"ID":"1503f898-c417-4ef8-8a3f-a33eed038eb6","Type":"ContainerStarted","Data":"cf56e48eb63f36fd03b09c4f034ecf80f4a2fb42eefd00ffcdcb38588dbc7d25"} Oct 30 01:01:54.574386 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:54.574068 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:01:54.596324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:54.596295 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:54.617295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:54.617222 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" podStartSLOduration=2.61720136 podStartE2EDuration="2.61720136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:54.614056338 +0000 UTC m=+1652.526183480" watchObservedRunningTime="2026-10-30 01:01:54.61720136 +0000 UTC m=+1652.529328503" Oct 30 01:01:56.462435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:56.462404 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:01:56.500854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:56.500815 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:58.193857 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:58.193822 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:59.795994 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:01:59.795958 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:00.759846 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:02:00.759809 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:02:01.432833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:01.432793 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:03.214376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:03.214292 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:05.037487 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:05.037457 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:05.586481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:05.586452 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:02:07.002028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:06.999536 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:07.123455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:07.123423 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:07.242527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:07.242482 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:07.699186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:07.699153 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:08.201583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:08.201544 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:08.969731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:08.953587 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:09.092757 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:09.092679 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:09.209470 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:09.209422 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:10.698547 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:10.698512 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:10.949105 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:10.949022 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:12.740211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.740175 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx"] Oct 30 01:02:12.744584 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.744548 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.767933 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.767848 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx"] Oct 30 01:02:12.770925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.770897 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5f4f9b\"/\"openshift-service-ca.crt\"" Oct 30 01:02:12.778630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.771307 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5f4f9b\"/\"kube-root-ca.crt\"" Oct 30 01:02:12.778630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.772136 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5f4f9b\"/\"default-dockercfg-58r5w\"" Oct 30 01:02:12.778974 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.773741 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5f4f9b\"/\"test-model-5f4f9b-kserve-self-signed-certs\"" Oct 30 01:02:12.801065 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.801029 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:12.833991 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.833938 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-home\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.834188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.834060 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn8j4\" (UniqueName: \"kubernetes.io/projected/2ba7987b-ebd3-4345-9a36-997cdc5b2754-kube-api-access-hn8j4\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.834188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.834153 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tls-certs\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.834282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.834255 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-model-cache\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.834391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.834370 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tmp-dir\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.834449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.834428 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-dshm\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.936406 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.936355 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tls-certs\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.936631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.936430 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-model-cache\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.936631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.936501 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tmp-dir\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.936631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.936542 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-dshm\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.936631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.936627 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-home\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.936910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.936680 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn8j4\" (UniqueName: \"kubernetes.io/projected/2ba7987b-ebd3-4345-9a36-997cdc5b2754-kube-api-access-hn8j4\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.936910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.936857 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-model-cache\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.937030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.936994 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tmp-dir\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.937159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.937136 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-home\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.939188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.939161 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-dshm\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.939694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.939675 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tls-certs\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.959934 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.959888 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn8j4\" (UniqueName: \"kubernetes.io/projected/2ba7987b-ebd3-4345-9a36-997cdc5b2754-kube-api-access-hn8j4\") pod \"test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:12.988425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:12.983596 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:13.067149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:13.065856 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:13.186262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:13.181124 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:13.313648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:13.313606 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:13.494755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:13.494720 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx"] Oct 30 01:02:13.496504 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:02:13.496461 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ba7987b_ebd3_4345_9a36_997cdc5b2754.slice/crio-9cd241fcd5a069287b20ab9a923eabc3385a5f1b6b2a5511c194c5dcf932cf1c WatchSource:0}: Error finding container 9cd241fcd5a069287b20ab9a923eabc3385a5f1b6b2a5511c194c5dcf932cf1c: Status 404 returned error can't find the container with id 9cd241fcd5a069287b20ab9a923eabc3385a5f1b6b2a5511c194c5dcf932cf1c Oct 30 01:02:13.724099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:13.724061 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" event={"ID":"2ba7987b-ebd3-4345-9a36-997cdc5b2754","Type":"ContainerStarted","Data":"97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092"} Oct 30 01:02:13.724326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:13.724113 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" event={"ID":"2ba7987b-ebd3-4345-9a36-997cdc5b2754","Type":"ContainerStarted","Data":"9cd241fcd5a069287b20ab9a923eabc3385a5f1b6b2a5511c194c5dcf932cf1c"} Oct 30 01:02:13.764678 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:13.764561 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" podStartSLOduration=1.764543675 podStartE2EDuration="1.764543675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:13.76265035 +0000 UTC m=+1671.674777492" watchObservedRunningTime="2026-10-30 01:02:13.764543675 +0000 UTC m=+1671.676670817" Oct 30 01:02:14.066872 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:14.066827 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:14.908902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:14.908868 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:15.033559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:15.033384 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:15.536586 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:15.536550 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:16.169168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.169130 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:16.244449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.244408 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb"] Oct 30 01:02:16.248498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.248469 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:16.270174 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.270140 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb"] Oct 30 01:02:16.273422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.273391 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dd257f\"" Oct 30 01:02:16.273422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.273405 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dd257f-dockercfg-dfbjz\"" Oct 30 01:02:16.302124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.302090 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:16.343315 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.343275 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr"] Oct 30 01:02:16.347754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.347726 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:16.402963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.402350 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr"] Oct 30 01:02:16.460797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.456811 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn998\" (UniqueName: \"kubernetes.io/projected/35c7e888-3215-475a-9c2e-4b4b9a9450df-kube-api-access-vn998\") pod \"payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb\" (UID: \"35c7e888-3215-475a-9c2e-4b4b9a9450df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:16.460797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.456951 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35c7e888-3215-475a-9c2e-4b4b9a9450df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb\" (UID: \"35c7e888-3215-475a-9c2e-4b4b9a9450df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:16.460797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.457006 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr\" (UID: \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:16.460797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.457051 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqdj\" (UniqueName: \"kubernetes.io/projected/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-kube-api-access-gkqdj\") pod \"payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr\" (UID: \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:16.577733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.568605 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn998\" (UniqueName: \"kubernetes.io/projected/35c7e888-3215-475a-9c2e-4b4b9a9450df-kube-api-access-vn998\") pod \"payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb\" (UID: \"35c7e888-3215-475a-9c2e-4b4b9a9450df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:16.577733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.570347 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35c7e888-3215-475a-9c2e-4b4b9a9450df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb\" (UID: \"35c7e888-3215-475a-9c2e-4b4b9a9450df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:16.577733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.570419 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr\" (UID: \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:16.577733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.570469 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqdj\" (UniqueName: \"kubernetes.io/projected/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-kube-api-access-gkqdj\") pod \"payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr\" (UID: \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:16.577733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.572579 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35c7e888-3215-475a-9c2e-4b4b9a9450df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb\" (UID: \"35c7e888-3215-475a-9c2e-4b4b9a9450df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:16.577733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.574488 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr\" (UID: \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:16.603692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.599373 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn998\" (UniqueName: \"kubernetes.io/projected/35c7e888-3215-475a-9c2e-4b4b9a9450df-kube-api-access-vn998\") pod \"payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb\" (UID: \"35c7e888-3215-475a-9c2e-4b4b9a9450df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:16.626316 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.624493 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqdj\" (UniqueName: \"kubernetes.io/projected/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-kube-api-access-gkqdj\") pod \"payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr\" (UID: \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:16.665022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.664984 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:16.864265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:16.864233 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:17.043282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.043255 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr"] Oct 30 01:02:17.043608 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:02:17.043581 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc486bf7f_4d59_4ab1_9e75_ff2ce2b13b5f.slice/crio-d2161323c9d2158d246c199201b56b293e5024fb7a82217b0d494193678c51b5 WatchSource:0}: Error finding container d2161323c9d2158d246c199201b56b293e5024fb7a82217b0d494193678c51b5: Status 404 returned error can't find the container with id d2161323c9d2158d246c199201b56b293e5024fb7a82217b0d494193678c51b5 Oct 30 01:02:17.192456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.192427 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb"] Oct 30 01:02:17.194815 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:02:17.194781 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c7e888_3215_475a_9c2e_4b4b9a9450df.slice/crio-d3f64d741e8b1ff2bb86971e3139d501630c24baf4f65ec7b9de7b784fab5dde WatchSource:0}: Error finding container d3f64d741e8b1ff2bb86971e3139d501630c24baf4f65ec7b9de7b784fab5dde: Status 404 returned error can't find the container with id d3f64d741e8b1ff2bb86971e3139d501630c24baf4f65ec7b9de7b784fab5dde Oct 30 01:02:17.245169 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.245129 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:17.763899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.763853 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" event={"ID":"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f","Type":"ContainerStarted","Data":"c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f"} Oct 30 01:02:17.764181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.763920 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" event={"ID":"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f","Type":"ContainerStarted","Data":"d2161323c9d2158d246c199201b56b293e5024fb7a82217b0d494193678c51b5"} Oct 30 01:02:17.764181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.764127 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:17.765887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.765863 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" event={"ID":"35c7e888-3215-475a-9c2e-4b4b9a9450df","Type":"ContainerStarted","Data":"28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d"} Oct 30 01:02:17.766007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.765904 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" event={"ID":"35c7e888-3215-475a-9c2e-4b4b9a9450df","Type":"ContainerStarted","Data":"d3f64d741e8b1ff2bb86971e3139d501630c24baf4f65ec7b9de7b784fab5dde"} Oct 30 01:02:17.766007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.765929 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:17.803202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.803138 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" podStartSLOduration=1.8031207729999998 podStartE2EDuration="1.803120773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:17.800402027 +0000 UTC m=+1675.712529169" watchObservedRunningTime="2026-10-30 01:02:17.803120773 +0000 UTC m=+1675.715247914" Oct 30 01:02:17.838262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:17.838197 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" podStartSLOduration=1.8381769239999999 podStartE2EDuration="1.838176924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:17.836819069 +0000 UTC m=+1675.748946215" watchObservedRunningTime="2026-10-30 01:02:17.838176924 +0000 UTC m=+1675.750304067" Oct 30 01:02:18.845011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:18.844970 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:20.544507 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:20.544465 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:21.879455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:21.879419 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:22.732665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:22.732627 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:23.067289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:23.067248 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:23.081551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:23.081516 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:23.853245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:23.853211 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:02:24.543253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:24.543213 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:26.229207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:26.229176 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:27.835463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:27.835422 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:28.778493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:28.778462 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:02:28.778752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:28.778539 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:02:29.519135 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:29.519099 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:30.760953 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:02:30.760912 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:02:31.284559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:31.284505 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:32.887407 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:32.887365 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf"] Oct 30 01:02:32.893464 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:32.893438 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:32.915509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:32.915479 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf"] Oct 30 01:02:32.938050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:32.938001 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1da7b2-serving-cert\"" Oct 30 01:02:32.954543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:32.954506 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:32.954779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:32.954556 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-metrics-certs\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:32.954779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:32.954718 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vkz\" (UniqueName: \"kubernetes.io/projected/15dc04f8-79b8-41d7-8ada-df7ed4019560-kube-api-access-q9vkz\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:32.954779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:32.954760 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dc04f8-79b8-41d7-8ada-df7ed4019560-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:32.983039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:32.982998 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:33.056148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.056104 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:33.056148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.056142 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-metrics-certs\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:33.056416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.056206 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vkz\" (UniqueName: \"kubernetes.io/projected/15dc04f8-79b8-41d7-8ada-df7ed4019560-kube-api-access-q9vkz\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:33.056416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.056239 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dc04f8-79b8-41d7-8ada-df7ed4019560-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:33.056416 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:02:33.056258 4517 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-serving-cert: secret "maas-api-e2e-inf-b-1da7b2-serving-cert" not found Oct 30 01:02:33.056416 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:02:33.056328 4517 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls podName:15dc04f8-79b8-41d7-8ada-df7ed4019560 nodeName:}" failed. No retries permitted until 2026-10-30 01:02:33.556309139 +0000 UTC m=+1691.468436264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls") pod "maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" (UID: "15dc04f8-79b8-41d7-8ada-df7ed4019560") : secret "maas-api-e2e-inf-b-1da7b2-serving-cert" not found Oct 30 01:02:33.056991 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.056972 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dc04f8-79b8-41d7-8ada-df7ed4019560-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:33.059549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.059523 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-metrics-certs\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:33.101178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.100791 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:33.105556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.104830 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vkz\" (UniqueName: \"kubernetes.io/projected/15dc04f8-79b8-41d7-8ada-df7ed4019560-kube-api-access-q9vkz\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:33.359343 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.359303 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:33.564468 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.564424 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:33.567226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.567198 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls\") pod \"maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:33.813732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:33.813111 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:34.142852 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:34.142661 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf"] Oct 30 01:02:34.148404 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:02:34.148369 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15dc04f8_79b8_41d7_8ada_df7ed4019560.slice/crio-8c40f15450da7fc27dcf10cb5b93793d6592e5469e5329c7e3d472565882329c WatchSource:0}: Error finding container 8c40f15450da7fc27dcf10cb5b93793d6592e5469e5329c7e3d472565882329c: Status 404 returned error can't find the container with id 8c40f15450da7fc27dcf10cb5b93793d6592e5469e5329c7e3d472565882329c Oct 30 01:02:34.951952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:34.951889 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" event={"ID":"15dc04f8-79b8-41d7-8ada-df7ed4019560","Type":"ContainerStarted","Data":"c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1"} Oct 30 01:02:34.952144 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:34.951996 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:34.952144 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:34.952018 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" event={"ID":"15dc04f8-79b8-41d7-8ada-df7ed4019560","Type":"ContainerStarted","Data":"8c40f15450da7fc27dcf10cb5b93793d6592e5469e5329c7e3d472565882329c"} Oct 30 01:02:34.996818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:34.996760 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" podStartSLOduration=2.623003597 podStartE2EDuration="2.996744932s" totalImagesPullingTime="373.741335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:34.99255192 +0000 UTC m=+1692.904679064" watchObservedRunningTime="2026-10-30 01:02:34.996744932 +0000 UTC m=+1692.908872073" Oct 30 01:02:35.093035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:35.092996 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:37.289976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:37.289946 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:39.223286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:39.223247 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:40.966624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:40.966591 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:02:40.991186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:40.991141 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:41.258966 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:41.258874 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:41.798307 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:41.798269 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:42.209529 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:42.209419 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:42.360213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:42.360178 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:42.493693 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:42.493597 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:43.216773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:43.216737 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:44.516074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:44.516029 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:45.022179 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:45.022140 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:46.393514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:46.393222 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:46.516581 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:46.516539 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:46.808590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:46.808560 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:48.088063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.088012 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:48.187271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.187234 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn"] Oct 30 01:02:48.191837 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.191817 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:02:48.206766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.206586 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ae06fe\"" Oct 30 01:02:48.210070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.210035 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:48.221152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.221120 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn"] Oct 30 01:02:48.385727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.385608 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnsld\" (UniqueName: \"kubernetes.io/projected/32397089-ac22-4e3f-ba02-a642f2fb90c2-kube-api-access-lnsld\") pod \"payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn\" (UID: \"32397089-ac22-4e3f-ba02-a642f2fb90c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:02:48.385926 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.385895 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32397089-ac22-4e3f-ba02-a642f2fb90c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn\" (UID: \"32397089-ac22-4e3f-ba02-a642f2fb90c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:02:48.492298 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.492252 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnsld\" (UniqueName: \"kubernetes.io/projected/32397089-ac22-4e3f-ba02-a642f2fb90c2-kube-api-access-lnsld\") pod \"payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn\" (UID: \"32397089-ac22-4e3f-ba02-a642f2fb90c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:02:48.493081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.493051 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32397089-ac22-4e3f-ba02-a642f2fb90c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn\" (UID: \"32397089-ac22-4e3f-ba02-a642f2fb90c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:02:48.494131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.494107 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32397089-ac22-4e3f-ba02-a642f2fb90c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn\" (UID: \"32397089-ac22-4e3f-ba02-a642f2fb90c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:02:48.519494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.519424 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnsld\" (UniqueName: \"kubernetes.io/projected/32397089-ac22-4e3f-ba02-a642f2fb90c2-kube-api-access-lnsld\") pod \"payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn\" (UID: \"32397089-ac22-4e3f-ba02-a642f2fb90c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:02:48.807905 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:48.807859 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:02:49.004122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:49.004077 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:49.168339 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:49.168249 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn"] Oct 30 01:02:49.175660 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:02:49.175627 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32397089_ac22_4e3f_ba02_a642f2fb90c2.slice/crio-8429798f2c0f19c459445bda3cf57066431ff6506a34dc01ca8cad75fd29b371 WatchSource:0}: Error finding container 8429798f2c0f19c459445bda3cf57066431ff6506a34dc01ca8cad75fd29b371: Status 404 returned error can't find the container with id 8429798f2c0f19c459445bda3cf57066431ff6506a34dc01ca8cad75fd29b371 Oct 30 01:02:50.086920 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:50.086882 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" event={"ID":"32397089-ac22-4e3f-ba02-a642f2fb90c2","Type":"ContainerStarted","Data":"fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592"} Oct 30 01:02:50.087106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:50.086942 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" event={"ID":"32397089-ac22-4e3f-ba02-a642f2fb90c2","Type":"ContainerStarted","Data":"8429798f2c0f19c459445bda3cf57066431ff6506a34dc01ca8cad75fd29b371"} Oct 30 01:02:50.087186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:50.087139 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:02:50.128073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:50.128007 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" podStartSLOduration=2.127992054 podStartE2EDuration="2.127992054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:50.124184628 +0000 UTC m=+1708.036311773" watchObservedRunningTime="2026-10-30 01:02:50.127992054 +0000 UTC m=+1708.040119195" Oct 30 01:02:50.734907 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:50.734865 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:52.642488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:52.642448 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:54.403396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:54.403357 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:55.907430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:55.907387 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:56.018326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:56.018287 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:56.421308 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:56.421275 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:56.627338 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:56.627261 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:56.921681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:56.921647 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:57.034194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:57.034128 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:57.408140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:57.408112 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:58.685309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:58.685268 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:58.803273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:58.803239 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:58.921122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:02:58.921092 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:00.697056 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:00.696992 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:00.766084 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:03:00.766040 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:03:00.819809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:00.819775 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:01.035310 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:01.035284 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:01.105532 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:01.105503 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:03:01.737103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:01.737059 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:01.848028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:01.847978 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:02.940582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:02.940543 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:03.087397 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.087348 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:03.213457 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.213378 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:03.268151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.268116 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h"] Oct 30 01:03:03.273898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.273877 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.297442 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.296393 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h"] Oct 30 01:03:03.370060 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.370024 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5f4f9b-openshift-default-dockercfg-w9b79\"" Oct 30 01:03:03.456496 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.456461 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-credential-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.456786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.456757 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-certs\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.456927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.456814 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.456992 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.456942 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65166d57-ef62-4bab-8c93-3441539c3c21-istiod-ca-cert\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.457106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.457088 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-istio-token\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.457182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.457126 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-data\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.457243 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.457192 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdlb\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-kube-api-access-nqdlb\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.457243 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.457219 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-envoy\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.457351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.457288 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65166d57-ef62-4bab-8c93-3441539c3c21-istio-podinfo\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.558865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.558823 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65166d57-ef62-4bab-8c93-3441539c3c21-istiod-ca-cert\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.559117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.559095 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-istio-token\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.559363 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.559337 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-data\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.560320 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.560183 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdlb\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-kube-api-access-nqdlb\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.560320 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.559769 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65166d57-ef62-4bab-8c93-3441539c3c21-istiod-ca-cert\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.560320 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.559877 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-data\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.560320 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.560223 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-envoy\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.560651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.560418 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65166d57-ef62-4bab-8c93-3441539c3c21-istio-podinfo\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.560651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.560483 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-credential-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.560651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.560564 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-certs\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.560651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.560640 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.563762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.561090 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-credential-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.563762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.561137 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.563762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.561285 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-certs\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.565525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.565499 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-envoy\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.569136 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.569097 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65166d57-ef62-4bab-8c93-3441539c3c21-istio-podinfo\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.583732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.582514 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-istio-token\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.583732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.582669 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdlb\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-kube-api-access-nqdlb\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.596583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.596535 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:03.981360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.981260 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h"] Oct 30 01:03:03.986384 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:03:03.986352 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65166d57_ef62_4bab_8c93_3441539c3c21.slice/crio-1eddbf5d8f1c333dd619892c50802aaeb163e6cdae964d27628a11bcb10066c9 WatchSource:0}: Error finding container 1eddbf5d8f1c333dd619892c50802aaeb163e6cdae964d27628a11bcb10066c9: Status 404 returned error can't find the container with id 1eddbf5d8f1c333dd619892c50802aaeb163e6cdae964d27628a11bcb10066c9 Oct 30 01:03:03.989615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.989573 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:03:03.989750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.989657 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:03:03.989750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:03.989697 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:03:04.227158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:04.227107 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" event={"ID":"65166d57-ef62-4bab-8c93-3441539c3c21","Type":"ContainerStarted","Data":"963b00ccc8716f1cf0fc558095ad81b10c97e87f76b621dc441898875f83fd0d"} Oct 30 01:03:04.227369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:04.227192 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" event={"ID":"65166d57-ef62-4bab-8c93-3441539c3c21","Type":"ContainerStarted","Data":"1eddbf5d8f1c333dd619892c50802aaeb163e6cdae964d27628a11bcb10066c9"} Oct 30 01:03:04.284644 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:04.284581 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" podStartSLOduration=1.284561866 podStartE2EDuration="1.284561866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:04.278983799 +0000 UTC m=+1722.191110940" watchObservedRunningTime="2026-10-30 01:03:04.284561866 +0000 UTC m=+1722.196689007" Oct 30 01:03:04.601785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:04.597138 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:05.164738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:05.164282 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:05.599247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:05.599210 4517 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 01:03:05.599442 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:05.599278 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 01:03:06.597856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:06.597823 4517 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 01:03:06.598242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:06.597895 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 01:03:07.121636 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:07.121585 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:07.598149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:07.598110 4517 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 01:03:07.598650 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:07.598169 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 01:03:08.598131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:08.598094 4517 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 01:03:08.598398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:08.598372 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 01:03:08.827537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:08.827504 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:09.012054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:09.011943 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:09.133235 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:09.133207 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:09.389192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:09.389161 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:09.598225 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:09.598186 4517 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 01:03:09.598375 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:09.598248 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 01:03:09.888067 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:09.888036 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:10.487562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:10.487527 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:10.617237 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:10.617201 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:10.729972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:10.729944 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:10.935187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:10.935158 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:10.937083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:10.937060 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:10.939406 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:10.938579 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:03:12.782071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:12.782036 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:14.328941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:14.328912 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:14.451797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:14.451762 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:14.687761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:14.687627 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:15.637801 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:15.637756 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:16.215486 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.215447 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:16.306389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.306345 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r"] Oct 30 01:03:16.310885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.310861 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:16.330957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.330915 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r"] Oct 30 01:03:16.336417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.336384 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f96ede\"" Oct 30 01:03:16.336417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.336413 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f96ede-dockercfg-4sn5w\"" Oct 30 01:03:16.341218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.341190 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:16.397731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.397076 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk"] Oct 30 01:03:16.415494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.415439 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49xjw\" (UniqueName: \"kubernetes.io/projected/25469977-6f4f-4788-ad6a-1f6f91c84ccf-kube-api-access-49xjw\") pod \"payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r\" (UID: \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:16.417216 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.417187 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:16.431927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.431886 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25469977-6f4f-4788-ad6a-1f6f91c84ccf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r\" (UID: \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:16.432751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.432687 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk"] Oct 30 01:03:16.479789 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.476134 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:16.546332 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.546268 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25469977-6f4f-4788-ad6a-1f6f91c84ccf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r\" (UID: \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:16.548485 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.548444 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34572ac-cb49-4dfd-b400-248d93f99f68-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk\" (UID: \"d34572ac-cb49-4dfd-b400-248d93f99f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:16.549184 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.549161 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tcc\" (UniqueName: \"kubernetes.io/projected/d34572ac-cb49-4dfd-b400-248d93f99f68-kube-api-access-p4tcc\") pod \"payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk\" (UID: \"d34572ac-cb49-4dfd-b400-248d93f99f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:16.569417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.569381 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49xjw\" (UniqueName: \"kubernetes.io/projected/25469977-6f4f-4788-ad6a-1f6f91c84ccf-kube-api-access-49xjw\") pod \"payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r\" (UID: \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:16.623269 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.547422 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25469977-6f4f-4788-ad6a-1f6f91c84ccf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r\" (UID: \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:16.623269 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.612066 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49xjw\" (UniqueName: \"kubernetes.io/projected/25469977-6f4f-4788-ad6a-1f6f91c84ccf-kube-api-access-49xjw\") pod \"payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r\" (UID: \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:16.628573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.628035 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:16.727826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.725125 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34572ac-cb49-4dfd-b400-248d93f99f68-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk\" (UID: \"d34572ac-cb49-4dfd-b400-248d93f99f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:16.727826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.725180 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tcc\" (UniqueName: \"kubernetes.io/projected/d34572ac-cb49-4dfd-b400-248d93f99f68-kube-api-access-p4tcc\") pod \"payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk\" (UID: \"d34572ac-cb49-4dfd-b400-248d93f99f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:16.727826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.726220 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34572ac-cb49-4dfd-b400-248d93f99f68-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk\" (UID: \"d34572ac-cb49-4dfd-b400-248d93f99f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:16.747002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.746904 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tcc\" (UniqueName: \"kubernetes.io/projected/d34572ac-cb49-4dfd-b400-248d93f99f68-kube-api-access-p4tcc\") pod \"payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk\" (UID: \"d34572ac-cb49-4dfd-b400-248d93f99f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:16.923181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:16.923142 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:17.030977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:17.030165 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r"] Oct 30 01:03:17.038486 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:03:17.038404 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25469977_6f4f_4788_ad6a_1f6f91c84ccf.slice/crio-b26806f6b1fa42b8051ef35967c2c240467c439adc4aa7142751dd50bec9d139 WatchSource:0}: Error finding container b26806f6b1fa42b8051ef35967c2c240467c439adc4aa7142751dd50bec9d139: Status 404 returned error can't find the container with id b26806f6b1fa42b8051ef35967c2c240467c439adc4aa7142751dd50bec9d139 Oct 30 01:03:17.243506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:17.243473 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk"] Oct 30 01:03:17.244746 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:03:17.244700 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34572ac_cb49_4dfd_b400_248d93f99f68.slice/crio-e690dbb9f0236b15f4d81361670c9e98f4842b756eebb3581a204ab67119de33 WatchSource:0}: Error finding container e690dbb9f0236b15f4d81361670c9e98f4842b756eebb3581a204ab67119de33: Status 404 returned error can't find the container with id e690dbb9f0236b15f4d81361670c9e98f4842b756eebb3581a204ab67119de33 Oct 30 01:03:17.338185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:17.338143 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" event={"ID":"d34572ac-cb49-4dfd-b400-248d93f99f68","Type":"ContainerStarted","Data":"e690dbb9f0236b15f4d81361670c9e98f4842b756eebb3581a204ab67119de33"} Oct 30 01:03:17.339951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:17.339925 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" event={"ID":"25469977-6f4f-4788-ad6a-1f6f91c84ccf","Type":"ContainerStarted","Data":"d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582"} Oct 30 01:03:17.340088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:17.339980 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" event={"ID":"25469977-6f4f-4788-ad6a-1f6f91c84ccf","Type":"ContainerStarted","Data":"b26806f6b1fa42b8051ef35967c2c240467c439adc4aa7142751dd50bec9d139"} Oct 30 01:03:17.340170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:17.340149 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:17.382720 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:17.382641 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" podStartSLOduration=1.382621182 podStartE2EDuration="1.382621182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:17.376734469 +0000 UTC m=+1735.288861607" watchObservedRunningTime="2026-10-30 01:03:17.382621182 +0000 UTC m=+1735.294748322" Oct 30 01:03:18.291884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:18.291845 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:18.351076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:18.351037 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" event={"ID":"d34572ac-cb49-4dfd-b400-248d93f99f68","Type":"ContainerStarted","Data":"f23479882d2939b589a3309d3e58fe9378205d72e2ea8a4ea996b9238f19e87f"} Oct 30 01:03:18.362731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:18.351889 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:19.869054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:19.869020 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:21.462270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:21.462233 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:23.513266 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:23.513225 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:26.638285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:26.638244 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:27.388164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:27.388125 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:28.354690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:28.354662 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:03:28.402611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:28.402535 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" podStartSLOduration=12.402517766 podStartE2EDuration="12.402517766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:18.402366952 +0000 UTC m=+1736.314494092" watchObservedRunningTime="2026-10-30 01:03:28.402517766 +0000 UTC m=+1746.314644932" Oct 30 01:03:28.704817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:28.704661 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:29.361484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:29.361452 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:03:30.007503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.007472 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj"] Oct 30 01:03:30.013235 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.013210 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.039101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.039070 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj"] Oct 30 01:03:30.043051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.043009 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-780077\"/\"openshift-service-ca.crt\"" Oct 30 01:03:30.043051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.043051 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-780077\"/\"kube-root-ca.crt\"" Oct 30 01:03:30.043291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.043094 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-780077\"/\"default-dockercfg-stdzr\"" Oct 30 01:03:30.043291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.043188 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-780077\"/\"test-model-780077-kserve-self-signed-certs\"" Oct 30 01:03:30.066868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.066829 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-tmp-dir\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.067080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.066876 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd017f4b-0b2f-409b-a895-79c151564111-tls-certs\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.067080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.066964 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zft68\" (UniqueName: \"kubernetes.io/projected/cd017f4b-0b2f-409b-a895-79c151564111-kube-api-access-zft68\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.067080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.067031 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-home\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.067253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.067135 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-dshm\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.067253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.067169 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-model-cache\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.071811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.071765 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:30.168729 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.168672 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-dshm\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.169033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.169001 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-model-cache\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.169232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.169215 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-tmp-dir\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.169415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.169400 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd017f4b-0b2f-409b-a895-79c151564111-tls-certs\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.169613 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.169415 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-model-cache\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.169613 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.169579 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-tmp-dir\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.169856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.169836 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zft68\" (UniqueName: \"kubernetes.io/projected/cd017f4b-0b2f-409b-a895-79c151564111-kube-api-access-zft68\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.170025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.170009 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-home\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.170509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.170477 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-home\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.171676 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.171653 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-dshm\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.172333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.172312 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd017f4b-0b2f-409b-a895-79c151564111-tls-certs\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.191568 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.191530 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zft68\" (UniqueName: \"kubernetes.io/projected/cd017f4b-0b2f-409b-a895-79c151564111-kube-api-access-zft68\") pod \"test-model-780077-kserve-755c9cc6c7-rhsgj\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.344733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.344010 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:30.605813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.605662 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:30.772926 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:03:30.772881 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:03:30.788192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:30.788164 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj"] Oct 30 01:03:30.790292 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:03:30.790254 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd017f4b_0b2f_409b_a895_79c151564111.slice/crio-0e274d24e8289474761a5dd63479a16ed76f30a6c8233f83401aea2487ef8dd6 WatchSource:0}: Error finding container 0e274d24e8289474761a5dd63479a16ed76f30a6c8233f83401aea2487ef8dd6: Status 404 returned error can't find the container with id 0e274d24e8289474761a5dd63479a16ed76f30a6c8233f83401aea2487ef8dd6 Oct 30 01:03:31.516780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:31.516737 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" event={"ID":"cd017f4b-0b2f-409b-a895-79c151564111","Type":"ContainerStarted","Data":"d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6"} Oct 30 01:03:31.516971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:31.516797 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" event={"ID":"cd017f4b-0b2f-409b-a895-79c151564111","Type":"ContainerStarted","Data":"0e274d24e8289474761a5dd63479a16ed76f30a6c8233f83401aea2487ef8dd6"} Oct 30 01:03:31.557811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:31.557749 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podStartSLOduration=2.55772883 podStartE2EDuration="2.55772883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:31.553771042 +0000 UTC m=+1749.465898176" watchObservedRunningTime="2026-10-30 01:03:31.55772883 +0000 UTC m=+1749.469855971" Oct 30 01:03:32.345056 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:32.344975 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:32.519954 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:32.519910 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:34.290385 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:34.290345 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:35.914694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:35.914654 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:36.057167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:36.057129 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:36.220360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:36.220274 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:38.105615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:38.105577 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:38.219473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:38.219434 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:38.331371 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:38.331330 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:39.794843 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:39.794803 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:40.345392 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:40.345357 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:40.358115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:40.358084 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:40.604594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:40.604502 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:03:41.520469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:41.520433 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:43.531158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:43.531121 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:45.132522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:45.132489 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:46.828513 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:46.828468 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:48.640567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:48.640533 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:49.945912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:49.945877 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9"] Oct 30 01:03:49.950779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:49.950758 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:49.963800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:49.963781 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a07920-openshift-default-dockercfg-pc9lh\"" Oct 30 01:03:49.994289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:49.994237 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9"] Oct 30 01:03:50.024558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.024523 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.024829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.024612 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.024829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.024734 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wv2w\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-kube-api-access-8wv2w\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.024985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.024883 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.024985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.024965 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.025092 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.025012 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.025092 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.025056 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.025268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.025141 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.025268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.025200 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.126503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.126450 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wv2w\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-kube-api-access-8wv2w\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.126764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.126530 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.126764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.126578 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.126764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.126619 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.126759 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.126923 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.126989 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.127010 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.127027 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.127149 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.127238 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.127252 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.127277 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.127843 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.127816 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.130156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.130126 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.130969 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.130921 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.150534 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.150500 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.150901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.150864 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wv2w\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-kube-api-access-8wv2w\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.267354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.265985 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:50.451898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.451842 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:50.673322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.673279 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:50.745249 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.745221 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9"] Oct 30 01:03:50.747241 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:03:50.747216 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd451a3_1147_45c1_acc3_628f6fa6ac4e.slice/crio-2331dd8c8a7cd326f53f068a6c21cd3b26d96dc9e8bfd754e03befdf84ff1e3d WatchSource:0}: Error finding container 2331dd8c8a7cd326f53f068a6c21cd3b26d96dc9e8bfd754e03befdf84ff1e3d: Status 404 returned error can't find the container with id 2331dd8c8a7cd326f53f068a6c21cd3b26d96dc9e8bfd754e03befdf84ff1e3d Oct 30 01:03:50.750300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.750265 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:03:50.750390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.750333 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:03:50.750390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.750370 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:03:50.793804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:50.793124 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:51.736473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:51.736442 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" event={"ID":"bcd451a3-1147-45c1-acc3-628f6fa6ac4e","Type":"ContainerStarted","Data":"44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0"} Oct 30 01:03:51.736973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:51.736492 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" event={"ID":"bcd451a3-1147-45c1-acc3-628f6fa6ac4e","Type":"ContainerStarted","Data":"2331dd8c8a7cd326f53f068a6c21cd3b26d96dc9e8bfd754e03befdf84ff1e3d"} Oct 30 01:03:52.278733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:52.278352 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:52.290867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:52.288053 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:03:52.290867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:52.288115 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 01:03:52.650024 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:52.649981 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:53.271732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:53.270817 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:03:53.271732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:53.270886 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 01:03:54.272733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:54.271591 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:03:54.272733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:54.271646 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 01:03:54.525793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:54.517028 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:55.280734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:55.270079 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:03:55.280734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:55.270140 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 01:03:55.650540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:55.650507 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:56.115881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:56.115837 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:56.268734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:56.267070 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:03:56.268734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:56.267130 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 01:03:57.275356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:57.275324 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:57.276844 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:57.276820 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:57.277522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:57.277494 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:03:57.346070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:57.346013 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" podStartSLOduration=8.345998629 podStartE2EDuration="8.345998629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:51.796959787 +0000 UTC m=+1769.709086927" watchObservedRunningTime="2026-10-30 01:03:57.345998629 +0000 UTC m=+1775.258125831" Oct 30 01:03:58.061422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:58.060255 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:58.516860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:03:58.516752 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:00.034031 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:00.033997 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:00.761876 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:04:00.761834 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:04:01.607308 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:01.607233 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:03.266542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.266503 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:03.599700 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.599627 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv"] Oct 30 01:04:03.609463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.605377 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.625563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.625536 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv"] Oct 30 01:04:03.629415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.629308 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-820c04\"/\"default-dockercfg-7rswl\"" Oct 30 01:04:03.629415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.629346 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-820c04\"/\"openshift-service-ca.crt\"" Oct 30 01:04:03.629415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.629308 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-820c04\"/\"kube-root-ca.crt\"" Oct 30 01:04:03.629415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.629388 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-820c04\"/\"test-model-820c04-kserve-self-signed-certs\"" Oct 30 01:04:03.660501 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.660463 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:03.725304 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.725262 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-home\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.725531 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.725488 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tmp-dir\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.725604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.725548 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8lx\" (UniqueName: \"kubernetes.io/projected/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-kube-api-access-7c8lx\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.725693 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.725670 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-model-cache\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.725784 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.725742 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-dshm\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.725850 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.725825 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tls-certs\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.828097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.828052 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tmp-dir\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.828305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.828150 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8lx\" (UniqueName: \"kubernetes.io/projected/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-kube-api-access-7c8lx\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.828305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.828204 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-model-cache\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.828433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.828268 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-dshm\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.828493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.828474 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tls-certs\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.828551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.828505 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tmp-dir\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.828612 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.828545 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-model-cache\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.828669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.828608 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-home\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.828934 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.828911 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-home\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.830692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.830668 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-dshm\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.831061 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.831046 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tls-certs\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.852980 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.852722 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8lx\" (UniqueName: \"kubernetes.io/projected/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-kube-api-access-7c8lx\") pod \"test-model-820c04-kserve-6cc6cc7569-ttdkv\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:03.933203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:03.933163 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:04.392541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:04.392317 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv"] Oct 30 01:04:04.394140 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:04:04.394110 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7049336_fa54_49d4_9d60_4bd14f9b5ae2.slice/crio-3be97f3a9e0b0235c20b85d1dc2e4198acd8a166e2df3231d12cc53f6f3a1404 WatchSource:0}: Error finding container 3be97f3a9e0b0235c20b85d1dc2e4198acd8a166e2df3231d12cc53f6f3a1404: Status 404 returned error can't find the container with id 3be97f3a9e0b0235c20b85d1dc2e4198acd8a166e2df3231d12cc53f6f3a1404 Oct 30 01:04:04.922083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:04.922048 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" event={"ID":"a7049336-fa54-49d4-9d60-4bd14f9b5ae2","Type":"ContainerStarted","Data":"7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574"} Oct 30 01:04:04.922083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:04.922093 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" event={"ID":"a7049336-fa54-49d4-9d60-4bd14f9b5ae2","Type":"ContainerStarted","Data":"3be97f3a9e0b0235c20b85d1dc2e4198acd8a166e2df3231d12cc53f6f3a1404"} Oct 30 01:04:04.934457 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:04.934434 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:04.964045 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:04.963988 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" podStartSLOduration=1.9639720779999998 podStartE2EDuration="1.963972078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:04.959107505 +0000 UTC m=+1782.871234645" watchObservedRunningTime="2026-10-30 01:04:04.963972078 +0000 UTC m=+1782.876099218" Oct 30 01:04:05.164065 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:05.163888 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:05.283695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:05.282271 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:05.571552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:05.571517 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:07.019513 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:07.019477 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:08.744050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:08.744016 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:09.300232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:09.300189 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:10.556215 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:10.556173 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:10.708329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:10.708291 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:12.171882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:12.171841 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:13.934626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:13.934592 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:13.946819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:13.946798 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:13.953054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:13.953035 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:14.029557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:14.029517 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:04:15.894387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:15.894354 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:16.390569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:16.390534 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:18.223918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:18.223880 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:19.696741 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:19.696688 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:19.808237 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:19.808199 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:19.918512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:19.918474 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:21.817630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:21.817596 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:23.480787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:23.475734 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:25.017774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:25.017686 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:26.910593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:26.910548 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:27.552511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.552477 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn"] Oct 30 01:04:27.557478 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.557451 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.572293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.572260 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ae06fe-openshift-default-dockercfg-ccc5k\"" Oct 30 01:04:27.574414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.574383 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4"] Oct 30 01:04:27.579304 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.579276 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.588981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.588958 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn"] Oct 30 01:04:27.604749 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.604690 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:27.611490 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.611460 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4"] Oct 30 01:04:27.677988 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.677949 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.677988 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.677991 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.678218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678149 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.678218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678180 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6885190-f535-453a-bac8-1f067c2b6f78-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.678218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678204 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.678356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678265 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.678356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678282 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.678426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678370 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.678483 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678465 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.678523 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678499 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.678617 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678599 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6885190-f535-453a-bac8-1f067c2b6f78-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.678683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678646 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnqdg\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-kube-api-access-tnqdg\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.678793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678775 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.678843 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678828 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.678915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.678900 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.679031 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.679014 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.679070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.679046 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.679152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.679135 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s79cg\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-kube-api-access-s79cg\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.780591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.780547 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.780803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.780600 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.780803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.780675 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.780803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.780721 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.780803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.780778 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s79cg\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-kube-api-access-s79cg\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.781027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.780849 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.780879 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.780971 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.781027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.780997 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6885190-f535-453a-bac8-1f067c2b6f78-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781035 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781093 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.781247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781121 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.781247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781155 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781240 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.781512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781255 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.781512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781306 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781370 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.781681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781267 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781562 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781589 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.781681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781633 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6885190-f535-453a-bac8-1f067c2b6f78-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781681 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnqdg\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-kube-api-access-tnqdg\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781970 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781767 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.781970 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.781863 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.782243 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.782148 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.782307 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.782277 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.782839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.782808 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.782972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.782947 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6885190-f535-453a-bac8-1f067c2b6f78-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.784408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.784380 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.784633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.784606 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.784987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.784967 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.785410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.785387 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6885190-f535-453a-bac8-1f067c2b6f78-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.803091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.803017 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.803091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.803066 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.804001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.803970 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s79cg\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-kube-api-access-s79cg\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.805298 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.805276 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnqdg\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-kube-api-access-tnqdg\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:27.844786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.844737 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:27.876635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.876598 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:27.900289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:27.900263 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:28.211522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:28.211431 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn"] Oct 30 01:04:28.227670 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:04:28.227643 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f99f06_6df7_4e18_9154_e37dee7d6f70.slice/crio-09017c1d6a686424c9a371d5466119af7d25588f97b1c3977787c3f4810f0d1d WatchSource:0}: Error finding container 09017c1d6a686424c9a371d5466119af7d25588f97b1c3977787c3f4810f0d1d: Status 404 returned error can't find the container with id 09017c1d6a686424c9a371d5466119af7d25588f97b1c3977787c3f4810f0d1d Oct 30 01:04:28.231230 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:28.231189 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:04:28.231333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:28.231300 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:04:28.231388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:28.231347 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:04:28.262559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:28.262531 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4"] Oct 30 01:04:28.268378 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:04:28.268354 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6885190_f535_453a_bac8_1f067c2b6f78.slice/crio-8cbdbeb978676f22de59b15238be5306c7991b1bb2918de5f2787acc5d4c8f2e WatchSource:0}: Error finding container 8cbdbeb978676f22de59b15238be5306c7991b1bb2918de5f2787acc5d4c8f2e: Status 404 returned error can't find the container with id 8cbdbeb978676f22de59b15238be5306c7991b1bb2918de5f2787acc5d4c8f2e Oct 30 01:04:28.272512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:28.272468 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:04:28.272598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:28.272563 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:04:28.272649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:28.272607 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:04:28.812151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:28.812090 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:29.157086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.152523 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" event={"ID":"e6885190-f535-453a-bac8-1f067c2b6f78","Type":"ContainerStarted","Data":"1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a"} Oct 30 01:04:29.157894 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.157292 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" event={"ID":"e6885190-f535-453a-bac8-1f067c2b6f78","Type":"ContainerStarted","Data":"8cbdbeb978676f22de59b15238be5306c7991b1bb2918de5f2787acc5d4c8f2e"} Oct 30 01:04:29.157894 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.157333 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" event={"ID":"d1f99f06-6df7-4e18-9154-e37dee7d6f70","Type":"ContainerStarted","Data":"475347a3b6b814fb65cf25c9e9234b6db3a4ca532030284eda55487a2df87fae"} Oct 30 01:04:29.157894 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.157360 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" event={"ID":"d1f99f06-6df7-4e18-9154-e37dee7d6f70","Type":"ContainerStarted","Data":"09017c1d6a686424c9a371d5466119af7d25588f97b1c3977787c3f4810f0d1d"} Oct 30 01:04:29.231172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.230121 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" podStartSLOduration=2.230099816 podStartE2EDuration="2.230099816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:29.214483107 +0000 UTC m=+1807.126610300" watchObservedRunningTime="2026-10-30 01:04:29.230099816 +0000 UTC m=+1807.142226958" Oct 30 01:04:29.268729 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.267697 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" podStartSLOduration=2.267677782 podStartE2EDuration="2.267677782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:29.260962432 +0000 UTC m=+1807.173089574" watchObservedRunningTime="2026-10-30 01:04:29.267677782 +0000 UTC m=+1807.179804924" Oct 30 01:04:29.897733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.881482 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:29.897733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.884855 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:04:29.897733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.884911 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:04:29.905696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.905667 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:29.905982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.905866 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:04:29.906131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:29.906108 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 01:04:30.831980 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:04:30.831786 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:04:30.881918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:30.881863 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:04:30.882130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:30.881929 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:04:30.917734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:30.902087 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:04:30.917734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:30.902144 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 01:04:31.004736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:31.003820 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:31.882742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:31.882167 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:04:31.882742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:31.882231 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:04:31.904735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:31.900741 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:04:31.904735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:31.900804 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 01:04:32.024737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.014794 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg"] Oct 30 01:04:32.035269 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.035238 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.083731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.082961 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dd257f\"/\"default-dockercfg-vtz8q\"" Oct 30 01:04:32.083731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.083071 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dd257f\"/\"kube-root-ca.crt\"" Oct 30 01:04:32.108735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.091890 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-model-cache\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.108735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.091958 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2tc\" (UniqueName: \"kubernetes.io/projected/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-kube-api-access-bt2tc\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.108735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.092013 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tls-certs\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.108735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.092106 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-dshm\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.108735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.092186 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tmp-dir\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.108735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.092301 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-home\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.120734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.118379 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dd257f\"/\"openshift-service-ca.crt\"" Oct 30 01:04:32.120734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.118523 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dd257f\"/\"test-model-dd257f-kserve-self-signed-certs\"" Oct 30 01:04:32.132733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.130663 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:32.142789 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.138634 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg"] Oct 30 01:04:32.200742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.200430 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tls-certs\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.200742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.200541 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-dshm\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.200742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.200620 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tmp-dir\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.201085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.200753 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-home\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.201085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.200951 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-model-cache\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.201085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.201001 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2tc\" (UniqueName: \"kubernetes.io/projected/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-kube-api-access-bt2tc\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.218754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.218378 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tmp-dir\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.222742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.219036 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-home\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.222742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.219298 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-model-cache\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.330762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.328653 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tls-certs\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.403830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.342140 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-dshm\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.403830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.354432 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2tc\" (UniqueName: \"kubernetes.io/projected/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-kube-api-access-bt2tc\") pod \"test-model-dd257f-kserve-748c8c878c-xfffg\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.403830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.354998 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:32.956930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.940245 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:04:32.956930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:32.940315 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:04:33.064247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.010645 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:33.189779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.172486 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:04:33.278730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.254845 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 01:04:33.424778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.421837 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 01:04:33.430191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.430140 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:04:33.463390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.433099 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 01:04:33.642548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.642517 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg"] Oct 30 01:04:33.837592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.837347 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:33.882083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.882048 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:04:33.882349 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:33.882323 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:04:34.063770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:34.047082 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:04:34.063770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:34.047140 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 01:04:34.467791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:34.467626 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" event={"ID":"3dae4e81-bfe5-46bf-977c-a7fa6412c48a","Type":"ContainerStarted","Data":"aef5cf916476508e3850b7aa59480c0cab34dc52f14593b243fa9ade49f460a6"} Oct 30 01:04:34.467791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:34.467700 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" event={"ID":"3dae4e81-bfe5-46bf-977c-a7fa6412c48a","Type":"ContainerStarted","Data":"ca35d1fc4cd0f1bf6b8d956e7d27c6e07ca67260f268281303ebc636fab98b70"} Oct 30 01:04:34.635735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:34.634886 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:34.890731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:34.879118 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:04:34.890731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:34.879186 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:04:34.901729 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:34.901092 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:04:34.901729 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:34.901217 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 01:04:35.373079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:35.373032 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:35.882505 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:35.882476 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:35.882997 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:35.882971 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:35.883827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:35.883810 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:04:35.907382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:35.907356 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:35.907771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:35.907749 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:35.908626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:35.908607 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:04:35.945455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:35.945400 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" podStartSLOduration=4.945384025 podStartE2EDuration="4.945384025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:34.540315451 +0000 UTC m=+1812.452442591" watchObservedRunningTime="2026-10-30 01:04:35.945384025 +0000 UTC m=+1813.857511196" Oct 30 01:04:36.391459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:36.391376 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:37.911536 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:37.911478 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:38.273511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:38.273407 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:40.065520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:40.065418 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:41.706775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:41.706700 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:42.151667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:42.151632 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:42.262469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:42.262434 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:42.372761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:42.372721 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:42.386245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:42.386216 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:42.565261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:42.565232 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:04:43.061343 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:43.061310 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:43.754017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:43.753985 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:45.689934 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:45.689900 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:47.281093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:47.281056 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:48.295205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:48.295168 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:49.168597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:49.168542 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:50.928381 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:50.928329 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:52.569072 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:52.569007 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:54.248007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:54.247960 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:56.216618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:56.216578 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:58.046454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:58.046395 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:59.925855 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:04:59.925791 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:00.767832 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:05:00.767695 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:05:01.495103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:01.495074 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:01.934420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:01.934367 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:03.325688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:03.325641 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:04.886340 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:04.886288 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:06.463116 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:06.463069 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:08.183110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:08.183075 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:09.819271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:09.819231 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:11.446475 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:11.446438 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:12.720730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.719769 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj"] Oct 30 01:05:12.726808 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.726781 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:12.742745 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.733776 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj"] Oct 30 01:05:12.747520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.747496 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:12.787856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.787817 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj"] Oct 30 01:05:12.968731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.934479 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07671c93-8322-4ca8-a227-10163344a3b1-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:12.969147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.969123 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:12.969386 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.969366 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:12.969567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.969542 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:12.969841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.969823 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:12.969996 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.969979 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzjx6\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-kube-api-access-gzjx6\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:12.970195 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.970179 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07671c93-8322-4ca8-a227-10163344a3b1-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:12.977809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.973778 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41154ec-0a4b-4362-96c1-b381495cf243-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:12.994315 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.994290 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.005727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:12.994550 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.006076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.006046 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghrbq\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-kube-api-access-ghrbq\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.006225 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.006209 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.006378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.006363 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.010853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.010796 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.016955 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.016906 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41154ec-0a4b-4362-96c1-b381495cf243-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.037964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.037914 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.038273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.038235 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.053885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.053849 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.168732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.157462 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.190486 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.190420 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.213852 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.213812 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.214193 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.214173 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41154ec-0a4b-4362-96c1-b381495cf243-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.239559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.193203 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.239863 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.231054 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41154ec-0a4b-4362-96c1-b381495cf243-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.240032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.234069 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.240305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.240155 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.240541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.240520 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.241037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.240981 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.241222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.241197 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.243596 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.241677 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.243868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.243850 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07671c93-8322-4ca8-a227-10163344a3b1-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.245086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.245058 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.245921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.245900 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.248197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.248177 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.248698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.248682 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.250572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.250551 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzjx6\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-kube-api-access-gzjx6\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.251140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.251121 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07671c93-8322-4ca8-a227-10163344a3b1-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.251806 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.251788 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41154ec-0a4b-4362-96c1-b381495cf243-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.252668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.252649 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.252912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.252892 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.253055 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.253040 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghrbq\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-kube-api-access-ghrbq\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.253348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.252557 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41154ec-0a4b-4362-96c1-b381495cf243-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.253487 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.248564 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.253580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.244913 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07671c93-8322-4ca8-a227-10163344a3b1-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.253666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.253351 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.253773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.253663 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.253937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.246319 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.254051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.245479 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.254399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.254360 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.260421 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.260393 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07671c93-8322-4ca8-a227-10163344a3b1-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.266761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.262192 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj"] Oct 30 01:05:13.310760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.286876 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.310760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.304081 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghrbq\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-kube-api-access-ghrbq\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.310760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.304812 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzjx6\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-kube-api-access-gzjx6\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.310760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.307676 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:13.536909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.536871 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:13.544424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:13.544393 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:14.044374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.043041 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj"] Oct 30 01:05:14.108999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.106686 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj"] Oct 30 01:05:14.111004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.110067 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:14.119684 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.119635 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:14.120106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.120063 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:14.120352 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.120313 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:14.141631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.130642 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:14.141631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.130762 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:14.141631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.130819 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:14.304261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.304167 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" event={"ID":"07671c93-8322-4ca8-a227-10163344a3b1","Type":"ContainerStarted","Data":"bdebc1035ecbf4efab62666940b181cf95088091dde1d6e6ebe06cf845c80e8a"} Oct 30 01:05:14.307596 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:14.307516 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" event={"ID":"e41154ec-0a4b-4362-96c1-b381495cf243","Type":"ContainerStarted","Data":"cfaa907b7d26499c3f5648097cbd38ce9cf42fac347e0a4ce9d92b6bba7311dc"} Oct 30 01:05:15.320515 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.320474 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" event={"ID":"07671c93-8322-4ca8-a227-10163344a3b1","Type":"ContainerStarted","Data":"05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834"} Oct 30 01:05:15.324040 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.324007 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" event={"ID":"e41154ec-0a4b-4362-96c1-b381495cf243","Type":"ContainerStarted","Data":"1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef"} Oct 30 01:05:15.374574 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.373760 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" podStartSLOduration=3.373743077 podStartE2EDuration="3.373743077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:15.372670105 +0000 UTC m=+1853.284797248" watchObservedRunningTime="2026-10-30 01:05:15.373743077 +0000 UTC m=+1853.285870216" Oct 30 01:05:15.423460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.423398 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" podStartSLOduration=3.42337928 podStartE2EDuration="3.42337928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:15.418347506 +0000 UTC m=+1853.330474661" watchObservedRunningTime="2026-10-30 01:05:15.42337928 +0000 UTC m=+1853.335506422" Oct 30 01:05:15.542425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.538897 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:15.542425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.539045 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:05:15.542425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.539109 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 01:05:15.545901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.544728 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:15.554727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.551967 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:05:15.554727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:15.552044 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:05:16.052745 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:16.052245 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:16.209735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:16.209247 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:16.542696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:16.542201 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:05:16.542696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:16.542274 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 01:05:16.548523 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:16.548379 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:05:16.548523 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:16.548458 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:05:17.545731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:17.545621 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:05:17.545731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:17.545682 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:05:17.546343 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:17.546273 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:05:17.546343 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:17.546317 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 01:05:18.193736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:18.193165 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:18.357630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:18.357584 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:18.548792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:18.541185 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:05:18.548792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:18.541263 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 01:05:18.669986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:18.669893 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:05:18.669986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:18.669963 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:05:19.538627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:19.538100 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:05:19.538627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:19.538163 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 01:05:19.545109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:19.545069 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:05:19.545292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:19.545130 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:05:20.269690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:20.269595 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:20.538363 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:20.538329 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:05:20.538639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:20.538614 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 01:05:20.545125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:20.545095 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:05:20.545364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:20.545341 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:05:21.537839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:21.537796 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:05:21.538281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:21.537885 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 01:05:21.545403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:21.545372 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:05:21.545622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:21.545597 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:05:22.131902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:22.131866 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:22.543511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:22.543478 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:22.543981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:22.543961 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:22.544880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:22.544859 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:05:22.549830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:22.549811 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:22.550101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:22.550076 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:22.551039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:22.551022 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:05:23.939362 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:23.939325 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:24.455067 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.455031 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k"] Oct 30 01:05:24.461405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.461374 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.477259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.477237 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dd257f-openshift-default-dockercfg-cs8k9\"" Oct 30 01:05:24.494279 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.494247 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k"] Oct 30 01:05:24.549569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.549512 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-certs\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.550022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.549991 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032e0c30-da56-4326-b70e-1a9ffe43554f-istiod-ca-cert\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.550164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.550113 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.550164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.550157 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-token\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.550282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.550263 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqj25\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-kube-api-access-cqj25\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.550370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.550339 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-credential-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.550436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.550422 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-envoy\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.550504 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.550488 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-data\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.550561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.550523 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-podinfo\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.652542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.652505 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-envoy\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.652751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.652583 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-data\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.652751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.652614 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-podinfo\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.652751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.652699 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-certs\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.652964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.652937 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032e0c30-da56-4326-b70e-1a9ffe43554f-istiod-ca-cert\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.653130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.653102 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-data\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.653222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.653127 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.653222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.653115 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-certs\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.653222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.653206 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-token\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.653384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.653277 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqj25\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-kube-api-access-cqj25\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.653384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.653331 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-credential-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.653666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.653406 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.653797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.653775 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-credential-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.653903 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.653829 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032e0c30-da56-4326-b70e-1a9ffe43554f-istiod-ca-cert\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.655361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.655339 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-envoy\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.655512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.655495 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-podinfo\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.675681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.675655 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-token\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.675962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.675927 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqj25\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-kube-api-access-cqj25\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:24.777314 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:24.777223 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:25.108476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:25.108450 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k"] Oct 30 01:05:25.109557 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:05:25.109528 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032e0c30_da56_4326_b70e_1a9ffe43554f.slice/crio-09c2371243d461288c59a16408cddb88b7d8f14794577cb0f1e94d1d86a37580 WatchSource:0}: Error finding container 09c2371243d461288c59a16408cddb88b7d8f14794577cb0f1e94d1d86a37580: Status 404 returned error can't find the container with id 09c2371243d461288c59a16408cddb88b7d8f14794577cb0f1e94d1d86a37580 Oct 30 01:05:25.112548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:25.112514 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:25.112620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:25.112581 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:25.112620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:25.112615 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:25.556928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:25.556858 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" event={"ID":"032e0c30-da56-4326-b70e-1a9ffe43554f","Type":"ContainerStarted","Data":"7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9"} Oct 30 01:05:25.556928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:25.556932 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" event={"ID":"032e0c30-da56-4326-b70e-1a9ffe43554f","Type":"ContainerStarted","Data":"09c2371243d461288c59a16408cddb88b7d8f14794577cb0f1e94d1d86a37580"} Oct 30 01:05:25.561575 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:25.561520 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:25.777745 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:25.777695 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:26.798762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:26.790825 4517 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:05:26.798762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:26.790905 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:05:27.151602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:27.151511 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:27.367592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:27.367557 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:27.778188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:27.778152 4517 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:05:27.778536 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:27.778492 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:05:28.782456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:28.782419 4517 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:05:28.783090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:28.783055 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:05:28.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:28.981572 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:29.738018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:29.737977 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:29.782739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:29.782238 4517 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:05:29.782739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:29.782312 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:05:30.782736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:30.778114 4517 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:05:30.782736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:30.778175 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:05:30.786744 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:05:30.785186 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:05:31.770025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:31.769983 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:31.778227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:31.778190 4517 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:05:31.778373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:31.778260 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:05:32.785759 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:32.785722 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:32.786286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:32.786240 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:32.787059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:32.787030 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:05:32.833655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:32.833584 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" podStartSLOduration=8.833567286 podStartE2EDuration="8.833567286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:25.622515724 +0000 UTC m=+1863.534642867" watchObservedRunningTime="2026-10-30 01:05:32.833567286 +0000 UTC m=+1870.745694428" Oct 30 01:05:33.589675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:33.586547 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:35.170191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:35.170147 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:36.791163 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:36.791120 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:38.636525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.636489 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:38.655381 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.655342 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr"] Oct 30 01:05:38.662655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.662612 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:38.694936 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.694903 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:05:38.695289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.695269 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:05:38.701786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.698488 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr"] Oct 30 01:05:38.822433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.822393 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vd92\" (UniqueName: \"kubernetes.io/projected/43def177-1c54-4500-b25f-faecab3b2af0-kube-api-access-8vd92\") pod \"maas-api-revoke-sub-1d8ba649-rfkxr\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:38.822626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.822436 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d8ba649-rfkxr\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:38.822675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.822626 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d8ba649-rfkxr\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:38.924983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.924889 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vd92\" (UniqueName: \"kubernetes.io/projected/43def177-1c54-4500-b25f-faecab3b2af0-kube-api-access-8vd92\") pod \"maas-api-revoke-sub-1d8ba649-rfkxr\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:38.924983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.924937 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d8ba649-rfkxr\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:38.925208 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.925026 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d8ba649-rfkxr\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:38.925690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.925655 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d8ba649-rfkxr\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:38.927686 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.927660 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d8ba649-rfkxr\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:38.948699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:38.948665 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vd92\" (UniqueName: \"kubernetes.io/projected/43def177-1c54-4500-b25f-faecab3b2af0-kube-api-access-8vd92\") pod \"maas-api-revoke-sub-1d8ba649-rfkxr\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:39.069134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:39.069099 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:39.383413 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:05:39.383384 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43def177_1c54_4500_b25f_faecab3b2af0.slice/crio-42b2e03d79ba3b21eaa4020883b8190396e2e501c54ba69301e9ee3f48c58640 WatchSource:0}: Error finding container 42b2e03d79ba3b21eaa4020883b8190396e2e501c54ba69301e9ee3f48c58640: Status 404 returned error can't find the container with id 42b2e03d79ba3b21eaa4020883b8190396e2e501c54ba69301e9ee3f48c58640 Oct 30 01:05:39.384677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:39.384650 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr"] Oct 30 01:05:39.785510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:39.785466 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" event={"ID":"43def177-1c54-4500-b25f-faecab3b2af0","Type":"ContainerStarted","Data":"f13ae32064723e716f7015020b13dd3b12e6b39e41bcf6c63f131b6cab3923ab"} Oct 30 01:05:39.786025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:39.785574 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" event={"ID":"43def177-1c54-4500-b25f-faecab3b2af0","Type":"ContainerStarted","Data":"42b2e03d79ba3b21eaa4020883b8190396e2e501c54ba69301e9ee3f48c58640"} Oct 30 01:05:39.830516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:39.830460 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" podStartSLOduration=1.830444711 podStartE2EDuration="1.830444711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:39.829050382 +0000 UTC m=+1877.741177524" watchObservedRunningTime="2026-10-30 01:05:39.830444711 +0000 UTC m=+1877.742571851" Oct 30 01:05:40.248348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.248316 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r"] Oct 30 01:05:40.253119 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.253095 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.273779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.273746 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r"] Oct 30 01:05:40.350077 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.350045 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4qh7\" (UniqueName: \"kubernetes.io/projected/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-kube-api-access-p4qh7\") pod \"maas-api-revoke-sub-fd8ae29d-wpz8r\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.350270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.350222 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8ae29d-wpz8r\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.350550 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.350519 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8ae29d-wpz8r\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.452539 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.452508 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8ae29d-wpz8r\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.452746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.452630 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4qh7\" (UniqueName: \"kubernetes.io/projected/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-kube-api-access-p4qh7\") pod \"maas-api-revoke-sub-fd8ae29d-wpz8r\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.452746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.452696 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8ae29d-wpz8r\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.453407 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.453374 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8ae29d-wpz8r\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.455472 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.455429 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8ae29d-wpz8r\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.478660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.478630 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4qh7\" (UniqueName: \"kubernetes.io/projected/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-kube-api-access-p4qh7\") pod \"maas-api-revoke-sub-fd8ae29d-wpz8r\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.532491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.532399 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:40.575915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.575879 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:40.796241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.796205 4517 generic.go:356] "Generic (PLEG): container finished" podID="43def177-1c54-4500-b25f-faecab3b2af0" containerID="f13ae32064723e716f7015020b13dd3b12e6b39e41bcf6c63f131b6cab3923ab" exitCode=0 Oct 30 01:05:40.796807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.796305 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" event={"ID":"43def177-1c54-4500-b25f-faecab3b2af0","Type":"ContainerDied","Data":"f13ae32064723e716f7015020b13dd3b12e6b39e41bcf6c63f131b6cab3923ab"} Oct 30 01:05:40.972583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:40.972546 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r"] Oct 30 01:05:40.973845 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:05:40.973816 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331ad4b0_f70f_4cc6_ad0a_45d3b061a5e3.slice/crio-6dad5da17b7976866ab305b43c6c216deaf928b9b948d0364c1e3b7e6e8b4d6d WatchSource:0}: Error finding container 6dad5da17b7976866ab305b43c6c216deaf928b9b948d0364c1e3b7e6e8b4d6d: Status 404 returned error can't find the container with id 6dad5da17b7976866ab305b43c6c216deaf928b9b948d0364c1e3b7e6e8b4d6d Oct 30 01:05:41.805615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:41.805583 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" event={"ID":"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3","Type":"ContainerStarted","Data":"1bf830fd82f4e3f7c02ffc19d8c106be1f75e3b77ae59b691ac3d7c13df68c4c"} Oct 30 01:05:41.806075 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:41.805636 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" event={"ID":"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3","Type":"ContainerStarted","Data":"6dad5da17b7976866ab305b43c6c216deaf928b9b948d0364c1e3b7e6e8b4d6d"} Oct 30 01:05:41.846105 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:41.846042 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" podStartSLOduration=1.846021057 podStartE2EDuration="1.846021057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:41.845277353 +0000 UTC m=+1879.757404496" watchObservedRunningTime="2026-10-30 01:05:41.846021057 +0000 UTC m=+1879.758148200" Oct 30 01:05:42.007422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.007390 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:42.098230 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.098193 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-xks-ca\") pod \"43def177-1c54-4500-b25f-faecab3b2af0\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " Oct 30 01:05:42.098389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.098305 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-openshift-ca\") pod \"43def177-1c54-4500-b25f-faecab3b2af0\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " Oct 30 01:05:42.098389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.098347 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43def177-1c54-4500-b25f-faecab3b2af0-kube-api-access-8vd92\" (UniqueName: \"kubernetes.io/projected/43def177-1c54-4500-b25f-faecab3b2af0-kube-api-access-8vd92\") pod \"43def177-1c54-4500-b25f-faecab3b2af0\" (UID: \"43def177-1c54-4500-b25f-faecab3b2af0\") " Oct 30 01:05:42.100082 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.100055 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-openshift-ca" pod "43def177-1c54-4500-b25f-faecab3b2af0" (UID: "43def177-1c54-4500-b25f-faecab3b2af0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:42.103836 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.103808 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43def177-1c54-4500-b25f-faecab3b2af0-kube-api-access-8vd92" pod "43def177-1c54-4500-b25f-faecab3b2af0" (UID: "43def177-1c54-4500-b25f-faecab3b2af0"). InnerVolumeSpecName "kube-api-access-8vd92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:42.104001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.103983 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-xks-ca" pod "43def177-1c54-4500-b25f-faecab3b2af0" (UID: "43def177-1c54-4500-b25f-faecab3b2af0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:05:42.201561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.201524 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:42.201561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.201565 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43def177-1c54-4500-b25f-faecab3b2af0-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:42.201793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.201581 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vd92\" (UniqueName: \"kubernetes.io/projected/43def177-1c54-4500-b25f-faecab3b2af0-kube-api-access-8vd92\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:42.276792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.276756 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:42.815228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.815200 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" Oct 30 01:05:42.815694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.815234 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr" event={"ID":"43def177-1c54-4500-b25f-faecab3b2af0","Type":"ContainerDied","Data":"42b2e03d79ba3b21eaa4020883b8190396e2e501c54ba69301e9ee3f48c58640"} Oct 30 01:05:42.815694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.815293 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b2e03d79ba3b21eaa4020883b8190396e2e501c54ba69301e9ee3f48c58640" Oct 30 01:05:42.817376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.817347 4517 generic.go:356] "Generic (PLEG): container finished" podID="331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3" containerID="1bf830fd82f4e3f7c02ffc19d8c106be1f75e3b77ae59b691ac3d7c13df68c4c" exitCode=0 Oct 30 01:05:42.817526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:42.817409 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" event={"ID":"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3","Type":"ContainerDied","Data":"1bf830fd82f4e3f7c02ffc19d8c106be1f75e3b77ae59b691ac3d7c13df68c4c"} Oct 30 01:05:43.848172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:43.848115 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:44.104766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.104679 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:44.135676 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.135632 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-kube-api-access-p4qh7\" (UniqueName: \"kubernetes.io/projected/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-kube-api-access-p4qh7\") pod \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " Oct 30 01:05:44.136275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.136249 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-xks-ca\") pod \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " Oct 30 01:05:44.136354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.136316 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-openshift-ca\") pod \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\" (UID: \"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3\") " Oct 30 01:05:44.136791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.136674 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-openshift-ca" pod "331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3" (UID: "331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:44.137318 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.137297 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:44.138324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.138299 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-kube-api-access-p4qh7" pod "331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3" (UID: "331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3"). InnerVolumeSpecName "kube-api-access-p4qh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:44.138649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.138626 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-xks-ca" pod "331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3" (UID: "331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:05:44.238948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.238913 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4qh7\" (UniqueName: \"kubernetes.io/projected/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-kube-api-access-p4qh7\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:44.239255 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.238966 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:44.833997 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.833959 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" event={"ID":"331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3","Type":"ContainerDied","Data":"6dad5da17b7976866ab305b43c6c216deaf928b9b948d0364c1e3b7e6e8b4d6d"} Oct 30 01:05:44.833997 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.834015 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dad5da17b7976866ab305b43c6c216deaf928b9b948d0364c1e3b7e6e8b4d6d" Oct 30 01:05:44.834268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:44.833977 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r" Oct 30 01:05:45.538386 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.538338 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:45.691092 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.690903 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv"] Oct 30 01:05:45.692037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.691996 4517 cpu_manager.go:414] "containerMap: removing container" podUID="331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3" containerName="revoke-keys" Oct 30 01:05:45.692214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.692197 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3" containerName="revoke-keys" Oct 30 01:05:45.692408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.692386 4517 cpu_manager.go:414] "containerMap: removing container" podUID="43def177-1c54-4500-b25f-faecab3b2af0" containerName="revoke-keys" Oct 30 01:05:45.692408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.692405 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43def177-1c54-4500-b25f-faecab3b2af0" containerName="revoke-keys" Oct 30 01:05:45.692624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.692611 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="43def177-1c54-4500-b25f-faecab3b2af0" containerName="revoke-keys" Oct 30 01:05:45.692668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.692634 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3" containerName="revoke-keys" Oct 30 01:05:45.697402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.697373 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.720494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.720443 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv"] Oct 30 01:05:45.726754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.723844 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f96ede\"/\"openshift-service-ca.crt\"" Oct 30 01:05:45.726754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.723994 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f96ede\"/\"test-model-f96ede-kserve-self-signed-certs\"" Oct 30 01:05:45.726754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.724077 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f96ede\"/\"kube-root-ca.crt\"" Oct 30 01:05:45.726754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.724183 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f96ede\"/\"default-dockercfg-p9sn8\"" Oct 30 01:05:45.758775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.758728 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:45.766190 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.766160 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-tmp-dir\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.766361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.766280 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-model-cache\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.766432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.766368 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-dshm\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.766432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.766398 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-home\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.766926 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.766880 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb17712e-f20a-4522-9f77-0af54460a715-tls-certs\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.767187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.767166 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbz5l\" (UniqueName: \"kubernetes.io/projected/fb17712e-f20a-4522-9f77-0af54460a715-kube-api-access-kbz5l\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.869541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.869442 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb17712e-f20a-4522-9f77-0af54460a715-tls-certs\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.869541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.869511 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbz5l\" (UniqueName: \"kubernetes.io/projected/fb17712e-f20a-4522-9f77-0af54460a715-kube-api-access-kbz5l\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.869804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.869603 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-tmp-dir\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.869804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.869671 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-model-cache\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.869804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.869771 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-dshm\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.869971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.869804 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-home\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.870356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.870263 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-model-cache\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.870677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.870635 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-home\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.870822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.870670 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-tmp-dir\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.874804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.873054 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb17712e-f20a-4522-9f77-0af54460a715-tls-certs\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.875202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.875173 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-dshm\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:45.894838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:45.894806 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbz5l\" (UniqueName: \"kubernetes.io/projected/fb17712e-f20a-4522-9f77-0af54460a715-kube-api-access-kbz5l\") pod \"test-model-f96ede-kserve-77bb6d6497-qlsnv\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:46.031733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:46.031389 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:46.603141 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:46.603087 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv"] Oct 30 01:05:46.615493 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:05:46.615464 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb17712e_f20a_4522_9f77_0af54460a715.slice/crio-bea01db453414441da06c96fb59a4924f1ff25504e3e4f423cb7dd9234aa14e4 WatchSource:0}: Error finding container bea01db453414441da06c96fb59a4924f1ff25504e3e4f423cb7dd9234aa14e4: Status 404 returned error can't find the container with id bea01db453414441da06c96fb59a4924f1ff25504e3e4f423cb7dd9234aa14e4 Oct 30 01:05:46.856805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:46.856670 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" event={"ID":"fb17712e-f20a-4522-9f77-0af54460a715","Type":"ContainerStarted","Data":"edd5543c64c99a31273e285cd594028b211a7c0c3e74fd954237aafb602ebb77"} Oct 30 01:05:46.856805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:46.856744 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" event={"ID":"fb17712e-f20a-4522-9f77-0af54460a715","Type":"ContainerStarted","Data":"bea01db453414441da06c96fb59a4924f1ff25504e3e4f423cb7dd9234aa14e4"} Oct 30 01:05:46.900102 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:46.900045 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" podStartSLOduration=1.900030114 podStartE2EDuration="1.900030114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:46.896643697 +0000 UTC m=+1884.808770828" watchObservedRunningTime="2026-10-30 01:05:46.900030114 +0000 UTC m=+1884.812157255" Oct 30 01:05:47.041738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:47.039825 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:47.444103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:47.444062 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:49.026114 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:49.026073 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:50.688774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:50.685605 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:50.843967 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:50.843932 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:51.389396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:51.389358 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq"] Oct 30 01:05:51.389642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:51.389619 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" containerID="cri-o://c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b" gracePeriod=30 Oct 30 01:05:51.437456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:51.436782 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2"] Oct 30 01:05:51.437456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:51.437040 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" containerID="cri-o://6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5" gracePeriod=30 Oct 30 01:05:51.444844 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:51.444809 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st"] Oct 30 01:05:51.445073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:51.445047 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" containerID="cri-o://0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab" gracePeriod=30 Oct 30 01:05:52.481894 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:52.481855 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:54.258461 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:54.258413 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:56.032543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.032506 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:56.045990 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.045964 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:56.130683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.130638 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:56.733267 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.733228 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 01:05:56.820177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.820136 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44"] Oct 30 01:05:56.821764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.821568 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" Oct 30 01:05:56.821764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.821595 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" Oct 30 01:05:56.821978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.821955 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" Oct 30 01:05:56.833660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.833629 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:56.837321 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.837293 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44"] Oct 30 01:05:56.864813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.856942 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 01:05:56.871940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.871904 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-certs\") pod \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " Oct 30 01:05:56.872103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.871966 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-envoy\") pod \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " Oct 30 01:05:56.872103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872009 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istiod-ca-cert\") pod \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " Oct 30 01:05:56.872103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872060 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-kube-api-access-sl8lb\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-kube-api-access-sl8lb\") pod \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " Oct 30 01:05:56.872278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872174 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-data\") pod \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " Oct 30 01:05:56.872278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872233 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-socket\") pod \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " Oct 30 01:05:56.872278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872235 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-certs" pod "5218673d-d5e7-40c3-ae83-3e5897e3ed12" (UID: "5218673d-d5e7-40c3-ae83-3e5897e3ed12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:56.872434 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872369 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-podinfo\") pod \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " Oct 30 01:05:56.872495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872443 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-token\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-token\") pod \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " Oct 30 01:05:56.872551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872494 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-credential-socket\") pod \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\" (UID: \"5218673d-d5e7-40c3-ae83-3e5897e3ed12\") " Oct 30 01:05:56.872551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872505 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-socket" pod "5218673d-d5e7-40c3-ae83-3e5897e3ed12" (UID: "5218673d-d5e7-40c3-ae83-3e5897e3ed12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:56.872832 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872804 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-credential-socket" pod "5218673d-d5e7-40c3-ae83-3e5897e3ed12" (UID: "5218673d-d5e7-40c3-ae83-3e5897e3ed12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:56.872986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.872959 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istiod-ca-cert" pod "5218673d-d5e7-40c3-ae83-3e5897e3ed12" (UID: "5218673d-d5e7-40c3-ae83-3e5897e3ed12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:56.873291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.873256 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:56.873291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.873276 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:56.873291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.873283 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:56.873291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.873291 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:56.873540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.873504 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-data" pod "5218673d-d5e7-40c3-ae83-3e5897e3ed12" (UID: "5218673d-d5e7-40c3-ae83-3e5897e3ed12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:56.876118 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.875914 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-kube-api-access-sl8lb" pod "5218673d-d5e7-40c3-ae83-3e5897e3ed12" (UID: "5218673d-d5e7-40c3-ae83-3e5897e3ed12"). InnerVolumeSpecName "kube-api-access-sl8lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:56.876334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.876309 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-podinfo" pod "5218673d-d5e7-40c3-ae83-3e5897e3ed12" (UID: "5218673d-d5e7-40c3-ae83-3e5897e3ed12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:05:56.877429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.877252 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-envoy" pod "5218673d-d5e7-40c3-ae83-3e5897e3ed12" (UID: "5218673d-d5e7-40c3-ae83-3e5897e3ed12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:56.877429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.877404 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-token" pod "5218673d-d5e7-40c3-ae83-3e5897e3ed12" (UID: "5218673d-d5e7-40c3-ae83-3e5897e3ed12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:56.885968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.885940 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 01:05:56.935450 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.935418 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx"] Oct 30 01:05:56.936212 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.936191 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" Oct 30 01:05:56.936303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.936215 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" Oct 30 01:05:56.936303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.936233 4517 cpu_manager.go:414] "containerMap: removing container" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" Oct 30 01:05:56.936303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.936241 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" Oct 30 01:05:56.936443 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.936428 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerName="istio-proxy" Oct 30 01:05:56.936480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.936449 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerName="istio-proxy" Oct 30 01:05:56.940276 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.940213 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:56.956015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.955421 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-780077-openshift-default-dockercfg-5jlcn\"" Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984309 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-kube-api-access-cxrlq\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-kube-api-access-cxrlq\") pod \"2d47e978-79d1-4d55-a14a-31ec5c412222\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984360 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-credential-socket\") pod \"2d47e978-79d1-4d55-a14a-31ec5c412222\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984412 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istiod-ca-cert\") pod \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984468 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-certs\") pod \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984536 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-socket\") pod \"2d47e978-79d1-4d55-a14a-31ec5c412222\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984618 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-token\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-token\") pod \"2d47e978-79d1-4d55-a14a-31ec5c412222\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984683 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-certs\") pod \"2d47e978-79d1-4d55-a14a-31ec5c412222\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984731 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-credential-socket\") pod \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984781 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-envoy\") pod \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984812 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-token\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-token\") pod \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984885 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-data\") pod \"2d47e978-79d1-4d55-a14a-31ec5c412222\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.984916 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-kube-api-access-vcmkv\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-kube-api-access-vcmkv\") pod \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985025 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-podinfo\") pod \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985154 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d47e978-79d1-4d55-a14a-31ec5c412222-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47e978-79d1-4d55-a14a-31ec5c412222-istiod-ca-cert\") pod \"2d47e978-79d1-4d55-a14a-31ec5c412222\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985233 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-envoy\") pod \"2d47e978-79d1-4d55-a14a-31ec5c412222\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985259 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-data\") pod \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985288 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-podinfo\") pod \"2d47e978-79d1-4d55-a14a-31ec5c412222\" (UID: \"2d47e978-79d1-4d55-a14a-31ec5c412222\") " Oct 30 01:05:56.985730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985408 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-socket\") pod \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\" (UID: \"a7c730b4-39d7-4fc1-9b40-75d12dabc451\") " Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985748 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985853 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985938 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.985971 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986026 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986097 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986378 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkl7\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-kube-api-access-hnkl7\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986431 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986585 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986678 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986693 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986727 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986744 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl8lb\" (UniqueName: \"kubernetes.io/projected/5218673d-d5e7-40c3-ae83-3e5897e3ed12-kube-api-access-sl8lb\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.986757 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5218673d-d5e7-40c3-ae83-3e5897e3ed12-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.987435 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-certs" pod "2d47e978-79d1-4d55-a14a-31ec5c412222" (UID: "2d47e978-79d1-4d55-a14a-31ec5c412222"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.987626 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-credential-socket" pod "2d47e978-79d1-4d55-a14a-31ec5c412222" (UID: "2d47e978-79d1-4d55-a14a-31ec5c412222"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.987950 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istiod-ca-cert" pod "a7c730b4-39d7-4fc1-9b40-75d12dabc451" (UID: "a7c730b4-39d7-4fc1-9b40-75d12dabc451"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:56.996575 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-socket" pod "2d47e978-79d1-4d55-a14a-31ec5c412222" (UID: "2d47e978-79d1-4d55-a14a-31ec5c412222"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.001829 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-certs" pod "a7c730b4-39d7-4fc1-9b40-75d12dabc451" (UID: "a7c730b4-39d7-4fc1-9b40-75d12dabc451"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.005308 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d47e978-79d1-4d55-a14a-31ec5c412222-istiod-ca-cert" pod "2d47e978-79d1-4d55-a14a-31ec5c412222" (UID: "2d47e978-79d1-4d55-a14a-31ec5c412222"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.005540 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-data" pod "2d47e978-79d1-4d55-a14a-31ec5c412222" (UID: "2d47e978-79d1-4d55-a14a-31ec5c412222"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.010541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.005868 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-credential-socket" pod "a7c730b4-39d7-4fc1-9b40-75d12dabc451" (UID: "a7c730b4-39d7-4fc1-9b40-75d12dabc451"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.011815 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.010679 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-envoy" pod "a7c730b4-39d7-4fc1-9b40-75d12dabc451" (UID: "a7c730b4-39d7-4fc1-9b40-75d12dabc451"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.016643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.016317 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-kube-api-access-vcmkv" pod "a7c730b4-39d7-4fc1-9b40-75d12dabc451" (UID: "a7c730b4-39d7-4fc1-9b40-75d12dabc451"). InnerVolumeSpecName "kube-api-access-vcmkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:57.016643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.016461 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-token" pod "a7c730b4-39d7-4fc1-9b40-75d12dabc451" (UID: "a7c730b4-39d7-4fc1-9b40-75d12dabc451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:57.016866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.016693 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-socket" pod "a7c730b4-39d7-4fc1-9b40-75d12dabc451" (UID: "a7c730b4-39d7-4fc1-9b40-75d12dabc451"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.018397 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.017624 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-podinfo" pod "a7c730b4-39d7-4fc1-9b40-75d12dabc451" (UID: "a7c730b4-39d7-4fc1-9b40-75d12dabc451"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:05:57.018397 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.017669 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-data" pod "a7c730b4-39d7-4fc1-9b40-75d12dabc451" (UID: "a7c730b4-39d7-4fc1-9b40-75d12dabc451"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.019032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.018941 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx"] Oct 30 01:05:57.021535 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.021512 4517 generic.go:356] "Generic (PLEG): container finished" podID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" containerID="6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5" exitCode=0 Oct 30 01:05:57.021691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.021661 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" event={"ID":"a7c730b4-39d7-4fc1-9b40-75d12dabc451","Type":"ContainerDied","Data":"6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5"} Oct 30 01:05:57.021868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.021731 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" event={"ID":"a7c730b4-39d7-4fc1-9b40-75d12dabc451","Type":"ContainerDied","Data":"35acd924eea98501a617262ec8c8ec8ae1d1e0be161a968e61e5aea100b6d9ff"} Oct 30 01:05:57.021868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.021763 4517 scope.go:122] "RemoveContainer" containerID="6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5" Oct 30 01:05:57.022052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.021928 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2" Oct 30 01:05:57.025425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.025389 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-kube-api-access-cxrlq" pod "2d47e978-79d1-4d55-a14a-31ec5c412222" (UID: "2d47e978-79d1-4d55-a14a-31ec5c412222"). InnerVolumeSpecName "kube-api-access-cxrlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:57.026120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.026090 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-envoy" pod "2d47e978-79d1-4d55-a14a-31ec5c412222" (UID: "2d47e978-79d1-4d55-a14a-31ec5c412222"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:57.026288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.026267 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-podinfo" pod "2d47e978-79d1-4d55-a14a-31ec5c412222" (UID: "2d47e978-79d1-4d55-a14a-31ec5c412222"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:05:57.027956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.027935 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-token" pod "2d47e978-79d1-4d55-a14a-31ec5c412222" (UID: "2d47e978-79d1-4d55-a14a-31ec5c412222"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:57.034262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.034235 4517 generic.go:356] "Generic (PLEG): container finished" podID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerID="0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab" exitCode=0 Oct 30 01:05:57.035783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.034510 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" event={"ID":"5218673d-d5e7-40c3-ae83-3e5897e3ed12","Type":"ContainerDied","Data":"0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab"} Oct 30 01:05:57.035783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.034854 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" event={"ID":"5218673d-d5e7-40c3-ae83-3e5897e3ed12","Type":"ContainerDied","Data":"fdea1aa6bd6fd181df4ef9c9dbc23d6847f5a82a0ae8c160ca6505267ae67fd4"} Oct 30 01:05:57.035783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.034888 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz"] Oct 30 01:05:57.036948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.034582 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" Oct 30 01:05:57.042091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.042068 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.054194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.053931 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz"] Oct 30 01:05:57.058034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.058012 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-969b92-openshift-default-dockercfg-4fk78\"" Oct 30 01:05:57.065749 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.062734 4517 generic.go:356] "Generic (PLEG): container finished" podID="2d47e978-79d1-4d55-a14a-31ec5c412222" containerID="c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b" exitCode=0 Oct 30 01:05:57.065749 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.063816 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" Oct 30 01:05:57.065749 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.064778 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" event={"ID":"2d47e978-79d1-4d55-a14a-31ec5c412222","Type":"ContainerDied","Data":"c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b"} Oct 30 01:05:57.065749 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.064834 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq" event={"ID":"2d47e978-79d1-4d55-a14a-31ec5c412222","Type":"ContainerDied","Data":"2d1d96636493845f4c31c7094d76ada617f5910d9b622e6ba524d5bd687af56f"} Oct 30 01:05:57.069609 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.069585 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:05:57.069878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.069633 4517 scope.go:122] "RemoveContainer" containerID="6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5" Oct 30 01:05:57.074381 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:05:57.074351 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5\": container with ID starting with 6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5 not found: ID does not exist" containerID="6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5" Oct 30 01:05:57.074550 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.074522 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5"} err="failed to get container status \"6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5\": rpc error: code = NotFound desc = could not find container \"6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5\": container with ID starting with 6f9c274722493ca49922a5d5f696e092a14b2e0d2546b9cb63e25478823b81f5 not found: ID does not exist" Oct 30 01:05:57.074669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.074655 4517 scope.go:122] "RemoveContainer" containerID="0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab" Oct 30 01:05:57.094408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094371 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-token\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.094780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094423 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtn7\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-kube-api-access-4vtn7\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.094780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094640 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.094780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094688 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.094780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094741 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-credential-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094798 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-envoy\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094828 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094880 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094911 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.094960 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095030 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095079 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095184 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-data\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095234 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16daf69b-dcf3-4a8e-9090-640ff89595f0-istiod-ca-cert\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095265 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-podinfo\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095316 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-certs\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095373 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkl7\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-kube-api-access-hnkl7\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095406 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095472 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095483 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxrlq\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-kube-api-access-cxrlq\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095492 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.095495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095501 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095514 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095523 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095531 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095539 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095547 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095555 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095563 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095571 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095579 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmkv\" (UniqueName: \"kubernetes.io/projected/a7c730b4-39d7-4fc1-9b40-75d12dabc451-kube-api-access-vcmkv\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095592 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095601 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d47e978-79d1-4d55-a14a-31ec5c412222-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095611 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095621 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c730b4-39d7-4fc1-9b40-75d12dabc451-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.095632 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d47e978-79d1-4d55-a14a-31ec5c412222-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.096426 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.097418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.096886 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.102904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.102876 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.103422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.103402 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.103641 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.103460 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.103753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.103467 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.106913 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.106892 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.116342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.116311 4517 scope.go:122] "RemoveContainer" containerID="0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab" Oct 30 01:05:57.116797 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:05:57.116771 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab\": container with ID starting with 0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab not found: ID does not exist" containerID="0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab" Oct 30 01:05:57.116900 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.116810 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab"} err="failed to get container status \"0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab\": rpc error: code = NotFound desc = could not find container \"0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab\": container with ID starting with 0dbd132b04717225a4137625dd92e177f87a6ad889311390fa180c1c722bf4ab not found: ID does not exist" Oct 30 01:05:57.116900 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.116837 4517 scope.go:122] "RemoveContainer" containerID="c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b" Oct 30 01:05:57.120433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.120406 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.121133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.121096 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkl7\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-kube-api-access-hnkl7\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.129177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.129118 4517 scope.go:122] "RemoveContainer" containerID="c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b" Oct 30 01:05:57.129458 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:05:57.129440 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b\": container with ID starting with c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b not found: ID does not exist" containerID="c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b" Oct 30 01:05:57.129539 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.129466 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b"} err="failed to get container status \"c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b\": rpc error: code = NotFound desc = could not find container \"c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b\": container with ID starting with c1ea0f94f74a5a15e2ef7b7535b014f6c54ce8bdd39928c0b19a5e20cc57ff2b not found: ID does not exist" Oct 30 01:05:57.169142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.169107 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:57.197956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.197870 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.198255 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.198223 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-data\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.198433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.198407 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16daf69b-dcf3-4a8e-9090-640ff89595f0-istiod-ca-cert\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.198582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.198563 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1a870c-347f-4068-b738-c137c098d791-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.198740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.198702 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-podinfo\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.198943 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.198909 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-certs\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.199103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.199083 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.199258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.199239 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.199447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.199413 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.199605 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.199586 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-token\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.199779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.199762 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtn7\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-kube-api-access-4vtn7\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.199924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.199906 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.200041 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.199943 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-certs\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.200196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.198593 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-data\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.200294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.200095 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1a870c-347f-4068-b738-c137c098d791-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.200434 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.200417 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.201182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.199280 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16daf69b-dcf3-4a8e-9090-640ff89595f0-istiod-ca-cert\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.201182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.200604 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-credential-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.201182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.198336 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.201182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.200692 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-envoy\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.201182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.200850 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxrw\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-kube-api-access-gbxrw\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.201182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.200899 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-credential-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.201182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.200899 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.211427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.211343 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-envoy\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.212412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.212386 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-podinfo\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.215589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.215563 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2"] Oct 30 01:05:57.225609 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.225541 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-token\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.226653 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.226632 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-ptzw2"] Oct 30 01:05:57.227423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.227405 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtn7\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-kube-api-access-4vtn7\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.246603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.246537 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st"] Oct 30 01:05:57.254762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.253536 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st"] Oct 30 01:05:57.259448 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.259426 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.305185 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.305255 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1a870c-347f-4068-b738-c137c098d791-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.305287 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.305467 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxrw\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-kube-api-access-gbxrw\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.305509 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.305650 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1a870c-347f-4068-b738-c137c098d791-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.305734 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.305764 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.305818 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.306302 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.306536 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.307302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.307169 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1a870c-347f-4068-b738-c137c098d791-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.314766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.314273 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1a870c-347f-4068-b738-c137c098d791-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.314766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.314543 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.314959 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.314792 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.315028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.314993 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.315088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.315044 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq"] Oct 30 01:05:57.319835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.319809 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-wfwdq"] Oct 30 01:05:57.333357 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.333297 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.339426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.339396 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxrw\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-kube-api-access-gbxrw\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.410737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.410520 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:57.524512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.524461 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44"] Oct 30 01:05:57.535117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.534696 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:57.535241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.535174 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:57.535241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.535226 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:57.637424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.637400 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx"] Oct 30 01:05:57.643836 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:05:57.643794 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16daf69b_dcf3_4a8e_9090_640ff89595f0.slice/crio-095d71c5ba0e7d3f1d43328e685404368a3ddde81ad23b3a7f2a36342ac4007f WatchSource:0}: Error finding container 095d71c5ba0e7d3f1d43328e685404368a3ddde81ad23b3a7f2a36342ac4007f: Status 404 returned error can't find the container with id 095d71c5ba0e7d3f1d43328e685404368a3ddde81ad23b3a7f2a36342ac4007f Oct 30 01:05:57.647506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.647464 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:57.647601 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.647563 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:57.647647 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.647615 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:57.685853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.685497 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:05:57.685853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.685552 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-5t7st" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:05:57.753639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.753601 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:57.812265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.812229 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz"] Oct 30 01:05:57.862489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.862427 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:57.862802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.862773 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:57.863081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:57.863055 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:05:58.093414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.093317 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" event={"ID":"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f","Type":"ContainerStarted","Data":"dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9"} Oct 30 01:05:58.094482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.094457 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" event={"ID":"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f","Type":"ContainerStarted","Data":"2057c7a09f26489c1a0c2b6b31fe9fe0f879187f3dfd2fdcfcb43c2d9a3d14c1"} Oct 30 01:05:58.098196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.098173 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" event={"ID":"16daf69b-dcf3-4a8e-9090-640ff89595f0","Type":"ContainerStarted","Data":"a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb"} Oct 30 01:05:58.098393 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.098377 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" event={"ID":"16daf69b-dcf3-4a8e-9090-640ff89595f0","Type":"ContainerStarted","Data":"095d71c5ba0e7d3f1d43328e685404368a3ddde81ad23b3a7f2a36342ac4007f"} Oct 30 01:05:58.101666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.101646 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" event={"ID":"3d1a870c-347f-4068-b738-c137c098d791","Type":"ContainerStarted","Data":"cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f"} Oct 30 01:05:58.101902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.101871 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" event={"ID":"3d1a870c-347f-4068-b738-c137c098d791","Type":"ContainerStarted","Data":"b82cf0bfb9abb1776729e9004b0d9bb8d23fe04ad7d47163901d023a3e83ba4d"} Oct 30 01:05:58.153019 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.152942 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podStartSLOduration=23.152927117 podStartE2EDuration="23.152927117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:58.144418372 +0000 UTC m=+1896.056545506" watchObservedRunningTime="2026-10-30 01:05:58.152927117 +0000 UTC m=+1896.065054257" Oct 30 01:05:58.169839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.169806 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:05:58.172292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.172171 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:05:58.172292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.172226 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:05:58.197890 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.197834 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podStartSLOduration=14.197820046 podStartE2EDuration="14.197820046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:58.187011677 +0000 UTC m=+1896.099138821" watchObservedRunningTime="2026-10-30 01:05:58.197820046 +0000 UTC m=+1896.109947186" Oct 30 01:05:58.235334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.232868 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podStartSLOduration=13.232850787 podStartE2EDuration="13.232850787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:58.231477679 +0000 UTC m=+1896.143604834" watchObservedRunningTime="2026-10-30 01:05:58.232850787 +0000 UTC m=+1896.144977931" Oct 30 01:05:58.246537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.246505 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d47e978-79d1-4d55-a14a-31ec5c412222" path="/var/lib/kubelet/pods/2d47e978-79d1-4d55-a14a-31ec5c412222/volumes" Oct 30 01:05:58.247498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.247466 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5218673d-d5e7-40c3-ae83-3e5897e3ed12" path="/var/lib/kubelet/pods/5218673d-d5e7-40c3-ae83-3e5897e3ed12/volumes" Oct 30 01:05:58.248422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.248401 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c730b4-39d7-4fc1-9b40-75d12dabc451" path="/var/lib/kubelet/pods/a7c730b4-39d7-4fc1-9b40-75d12dabc451/volumes" Oct 30 01:05:58.260033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.260010 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:05:58.261980 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.261938 4517 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:05:58.262151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.262128 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:05:58.411211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:58.411128 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:05:59.178182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:59.178142 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:05:59.187850 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:59.187796 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:05:59.265739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:59.262182 4517 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:05:59.265739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:59.262257 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:05:59.427160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:59.424851 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:05:59.427160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:59.424920 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:05:59.428957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:05:59.428662 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:00.185406 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.185368 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:06:00.186030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.185998 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:06:00.279226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.279182 4517 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:06:00.279986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.279541 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:06:00.416348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.416295 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:06:00.416645 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.416595 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:06:00.420547 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.420509 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 01:06:00.421191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.421163 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:00.463795 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.463204 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:00.591734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.590832 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:00.611734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.599800 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:00.728956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.728844 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:00.729807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.729770 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:00.904347 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:06:00.904271 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:06:00.968808 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:00.968103 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:01.028925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.028791 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:01.060455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.058451 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:01.060769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.060747 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:01.062353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.062316 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:01.075763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.075422 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": context deadline exceeded" Oct 30 01:06:01.177459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.170272 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:06:01.177459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.170440 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:06:01.256194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.255048 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 01:06:01.260097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.260068 4517 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:06:01.260476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.260449 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:06:01.362019 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.361063 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:01.416831 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.416797 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:06:01.417109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.417085 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:06:01.423818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.423781 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:01.424398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.424378 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 01:06:01.426776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.426751 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:01.427234 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.427218 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 01:06:01.458695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.458657 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:01.598878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.598827 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:01.599568 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:01.599533 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:02.077735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.068411 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:02.100742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.079363 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 01:06:02.100742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.082035 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:02.100742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.100116 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:06:02.305845 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.305807 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 01:06:02.392506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.392418 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:06:02.393206 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.393165 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:06:02.430818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.430776 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 01:06:02.453059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.453003 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:02.567651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.567608 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 01:06:02.688822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.660059 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 01:06:02.722681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.722634 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:02.734668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.734623 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:02.748914 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.748868 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": context deadline exceeded" Oct 30 01:06:02.773006 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.772960 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:02.774725 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.774673 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:06:02.795005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.794964 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:06:02.834254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.798387 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 01:06:02.838982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.838949 4517 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:06:02.839230 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.839200 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:06:02.857681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:02.857616 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:03.032968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.032855 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:03.063476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.063437 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" podUID="a7049336-fa54-49d4-9d60-4bd14f9b5ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:03.087583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.085365 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 01:06:03.101615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.101309 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 01:06:03.101615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.101453 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:06:03.101615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.101498 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:06:03.101615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.101538 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 01:06:03.120776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.118933 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wmm7r" podUID="23dedc80-2107-4d18-be9f-85405fcfa02c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:03.164841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.164788 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": context deadline exceeded" Oct 30 01:06:03.175870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.174993 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:06:03.175870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.175054 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:06:03.212953 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.212230 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 01:06:03.294481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.294071 4517 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:06:03.294481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.294129 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:06:03.377732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.374202 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:06:03.439178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.436483 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:03.439178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.439071 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:06:03.439178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.439115 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:06:03.482590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.477290 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:03.548945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.548833 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:03.604123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.601296 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:03.639762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.636849 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:03.777795 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.773826 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:03.777795 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.774543 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:06:03.777795 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.774889 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 01:06:03.780310 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.778806 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": context deadline exceeded" Oct 30 01:06:03.780310 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.779102 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": context deadline exceeded" Oct 30 01:06:03.922833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.914958 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 01:06:03.922833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:03.916074 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.041121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.041079 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 01:06:04.044380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.044354 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 01:06:04.066633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.058277 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.097261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.097222 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.097881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.097857 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 01:06:04.330733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.329781 4517 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:06:04.330733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.329840 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:06:04.371979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.369038 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 30 01:06:04.371979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.370861 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:06:04.371979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.370911 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:06:04.444557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.442869 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:06:04.444557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.442931 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:06:04.446288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.445436 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.485188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.484791 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.485188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.484885 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 01:06:04.571349 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.571311 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 01:06:04.633186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.633098 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:04.662071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.662021 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.678747 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.663076 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 01:06:04.782064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.782022 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.782581 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.782550 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": context deadline exceeded" Oct 30 01:06:04.786684 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.776804 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.801236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.792887 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 01:06:04.828595 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.828549 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.908990 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.908888 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" podUID="2ba7987b-ebd3-4345-9a36-997cdc5b2754" containerName="main" probeResult="failure" output="Get \"https://10.133.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:04.931727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.931607 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:05.000739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:04.992493 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 01:06:05.026551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.026511 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" podUID="a7049336-fa54-49d4-9d60-4bd14f9b5ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:05.069444 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.053500 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:05.125177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.125121 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 01:06:05.251884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.206922 4517 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:06:05.252178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.252145 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:06:05.287979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.287934 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:05.295747 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.294089 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 01:06:05.337752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.322813 4517 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:06:05.337752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.322881 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:06:05.337752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.329800 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:05.451753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.419331 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 01:06:05.574498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.574449 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:05.621201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.619214 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 01:06:05.642029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.641984 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 01:06:05.645785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.644580 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 01:06:05.696925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.696893 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:06:05.697741 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.697189 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:06:05.698786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.698766 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 01:06:05.716908 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.716871 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:05.781793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.750586 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 01:06:05.784835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.783619 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:05.797537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.785262 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:05.813070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.809299 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 01:06:05.951786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.946608 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:06:05.951786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.949379 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 01:06:05.951786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:05.949928 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:06:06.181671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.181643 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:06:06.183664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.183642 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:06:06.184958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.184935 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:06:06.324137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.324103 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:06:06.334963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.325432 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:06:06.334963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.326926 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:06:06.356005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.355973 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:06.427734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.423831 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb"] Oct 30 01:06:06.427734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.424149 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" containerID="cri-o://6428260b307b3e34249bd58d23ebbbaab46768466ce1f35914b1e0e580d7e10a" gracePeriod=30 Oct 30 01:06:06.522563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.521793 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm"] Oct 30 01:06:06.522563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.522024 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" containerID="cri-o://e2f5debeef1a62bfe7cf94905fedd5e002fc54b622bcfdc312ddd7a0274c9c64" gracePeriod=30 Oct 30 01:06:06.818039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.817956 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:06:06.820080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.819093 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:06:06.822762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:06.822370 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:06:07.895091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:07.895043 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:08.095722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:08.095670 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:09.616263 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:09.615480 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx"] Oct 30 01:06:09.659070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:09.659026 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:10.181042 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.181007 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:10.328043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.322840 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2"] Oct 30 01:06:10.328043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.327585 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2"] Oct 30 01:06:10.328043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.327700 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.405180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.405146 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:06:10.405180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.405182 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:06:10.493834 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.493739 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgltq\" (UniqueName: \"kubernetes.io/projected/9efb88a8-9e61-444f-b996-32948d51920b-kube-api-access-mgltq\") pod \"maas-api-revoke-sub-93d50254-449g2\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.494009 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.493852 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d50254-449g2\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.494086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.494004 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d50254-449g2\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.595122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.595089 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d50254-449g2\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.595323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.595178 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d50254-449g2\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.595405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.595391 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgltq\" (UniqueName: \"kubernetes.io/projected/9efb88a8-9e61-444f-b996-32948d51920b-kube-api-access-mgltq\") pod \"maas-api-revoke-sub-93d50254-449g2\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.595917 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.595893 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d50254-449g2\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.597652 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.597627 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d50254-449g2\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.619699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.619675 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgltq\" (UniqueName: \"kubernetes.io/projected/9efb88a8-9e61-444f-b996-32948d51920b-kube-api-access-mgltq\") pod \"maas-api-revoke-sub-93d50254-449g2\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.649598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.649567 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:10.969190 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:10.969161 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2"] Oct 30 01:06:10.969421 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:06:10.969387 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efb88a8_9e61_444f_b996_32948d51920b.slice/crio-e45952107f7f9403dbb287d8e0e2e61433d63e4402eb9a499b8599e516e82652 WatchSource:0}: Error finding container e45952107f7f9403dbb287d8e0e2e61433d63e4402eb9a499b8599e516e82652: Status 404 returned error can't find the container with id e45952107f7f9403dbb287d8e0e2e61433d63e4402eb9a499b8599e516e82652 Oct 30 01:06:12.003128 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.001041 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" event={"ID":"9efb88a8-9e61-444f-b996-32948d51920b","Type":"ContainerStarted","Data":"0a7ec28a4c7023868efedafd438d444504d7585404ea7f1b097893ccca887484"} Oct 30 01:06:12.003128 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.001110 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" event={"ID":"9efb88a8-9e61-444f-b996-32948d51920b","Type":"ContainerStarted","Data":"e45952107f7f9403dbb287d8e0e2e61433d63e4402eb9a499b8599e516e82652"} Oct 30 01:06:12.006542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.006511 4517 generic.go:356] "Generic (PLEG): container finished" podID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerID="6428260b307b3e34249bd58d23ebbbaab46768466ce1f35914b1e0e580d7e10a" exitCode=0 Oct 30 01:06:12.006862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.006843 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" event={"ID":"a8e883ea-a69a-4d3b-a80b-bc90b9edb040","Type":"ContainerDied","Data":"6428260b307b3e34249bd58d23ebbbaab46768466ce1f35914b1e0e580d7e10a"} Oct 30 01:06:12.010007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.009699 4517 generic.go:356] "Generic (PLEG): container finished" podID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerID="e2f5debeef1a62bfe7cf94905fedd5e002fc54b622bcfdc312ddd7a0274c9c64" exitCode=0 Oct 30 01:06:12.010007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.009939 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" event={"ID":"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd","Type":"ContainerDied","Data":"e2f5debeef1a62bfe7cf94905fedd5e002fc54b622bcfdc312ddd7a0274c9c64"} Oct 30 01:06:12.046863 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.043211 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" podStartSLOduration=2.04319286 podStartE2EDuration="2.04319286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:12.04173485 +0000 UTC m=+1909.953861989" watchObservedRunningTime="2026-10-30 01:06:12.04319286 +0000 UTC m=+1909.955320002" Oct 30 01:06:12.150639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.150608 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.331316 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-data\") pod \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.331363 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-podinfo\") pod \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.331445 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-kube-api-access-wpfwv\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-kube-api-access-wpfwv\") pod \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.331502 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-certs\") pod \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.331548 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-envoy\") pod \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.331691 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-credential-socket\") pod \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.331779 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-socket\") pod \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.331853 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istiod-ca-cert\") pod \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.331984 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-token\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-token\") pod \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\" (UID: \"a8e883ea-a69a-4d3b-a80b-bc90b9edb040\") " Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.332311 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-certs" pod "a8e883ea-a69a-4d3b-a80b-bc90b9edb040" (UID: "a8e883ea-a69a-4d3b-a80b-bc90b9edb040"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.332883 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.333407 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-data" pod "a8e883ea-a69a-4d3b-a80b-bc90b9edb040" (UID: "a8e883ea-a69a-4d3b-a80b-bc90b9edb040"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.335838 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.339811 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-credential-socket" pod "a8e883ea-a69a-4d3b-a80b-bc90b9edb040" (UID: "a8e883ea-a69a-4d3b-a80b-bc90b9edb040"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.379207 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-socket" pod "a8e883ea-a69a-4d3b-a80b-bc90b9edb040" (UID: "a8e883ea-a69a-4d3b-a80b-bc90b9edb040"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.386013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.385577 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istiod-ca-cert" pod "a8e883ea-a69a-4d3b-a80b-bc90b9edb040" (UID: "a8e883ea-a69a-4d3b-a80b-bc90b9edb040"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:12.419774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.410042 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-token" pod "a8e883ea-a69a-4d3b-a80b-bc90b9edb040" (UID: "a8e883ea-a69a-4d3b-a80b-bc90b9edb040"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.432198 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.433934 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-token\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-token\") pod \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.434080 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-data\") pod \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.434186 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-socket\") pod \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.434213 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-certs\") pod \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.434283 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-podinfo\") pod \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.434371 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-envoy\") pod \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.434410 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istiod-ca-cert\") pod \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.434450 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-kube-api-access-4tnt9\") pod \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.434615 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-credential-socket\") pod \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\" (UID: \"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd\") " Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.435513 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.435538 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.435552 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.435565 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.435579 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.445542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.437167 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-envoy" pod "a8e883ea-a69a-4d3b-a80b-bc90b9edb040" (UID: "a8e883ea-a69a-4d3b-a80b-bc90b9edb040"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.497021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.467526 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istiod-ca-cert" pod "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" (UID: "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:12.497021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.476341 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-certs" pod "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" (UID: "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.497021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.496862 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-data" pod "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" (UID: "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.497021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.496974 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-socket" pod "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" (UID: "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.527372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.499814 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-credential-socket" pod "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" (UID: "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.527372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.525945 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-kube-api-access-wpfwv" pod "a8e883ea-a69a-4d3b-a80b-bc90b9edb040" (UID: "a8e883ea-a69a-4d3b-a80b-bc90b9edb040"). InnerVolumeSpecName "kube-api-access-wpfwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:12.550201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.538502 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.550201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.538529 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.550201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.538544 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.550201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.538560 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.550201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.538576 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpfwv\" (UniqueName: \"kubernetes.io/projected/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-kube-api-access-wpfwv\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.550201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.538591 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.550201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.538604 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.550201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.546153 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-envoy" pod "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" (UID: "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.551001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.550867 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-podinfo" pod "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" (UID: "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:12.594752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.594519 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-podinfo" pod "a8e883ea-a69a-4d3b-a80b-bc90b9edb040" (UID: "a8e883ea-a69a-4d3b-a80b-bc90b9edb040"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:12.596233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.596205 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-token" pod "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" (UID: "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:12.603486 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.603451 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-kube-api-access-4tnt9" pod "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" (UID: "a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd"). InnerVolumeSpecName "kube-api-access-4tnt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:12.640505 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.640469 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.640505 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.640507 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e883ea-a69a-4d3b-a80b-bc90b9edb040-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.640962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.640524 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.640962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.640537 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tnt9\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-kube-api-access-4tnt9\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.640962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:12.640550 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.023335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.023236 4517 generic.go:356] "Generic (PLEG): container finished" podID="9efb88a8-9e61-444f-b996-32948d51920b" containerID="0a7ec28a4c7023868efedafd438d444504d7585404ea7f1b097893ccca887484" exitCode=0 Oct 30 01:06:13.023839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.023321 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" event={"ID":"9efb88a8-9e61-444f-b996-32948d51920b","Type":"ContainerDied","Data":"0a7ec28a4c7023868efedafd438d444504d7585404ea7f1b097893ccca887484"} Oct 30 01:06:13.025651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.025627 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" Oct 30 01:06:13.025810 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.025663 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb" event={"ID":"a8e883ea-a69a-4d3b-a80b-bc90b9edb040","Type":"ContainerDied","Data":"50388e6b5e002ab1c11327137bd74454d9cef9122a709c7adbdf09d39af35210"} Oct 30 01:06:13.025899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.025829 4517 scope.go:122] "RemoveContainer" containerID="6428260b307b3e34249bd58d23ebbbaab46768466ce1f35914b1e0e580d7e10a" Oct 30 01:06:13.038803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.038770 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" event={"ID":"a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd","Type":"ContainerDied","Data":"75441b73df22df9b1005621133736007164a7357709eba082982a21e47a9e35e"} Oct 30 01:06:13.038998 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.038979 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm" Oct 30 01:06:13.053891 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.051673 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:13.093241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.083697 4517 scope.go:122] "RemoveContainer" containerID="e2f5debeef1a62bfe7cf94905fedd5e002fc54b622bcfdc312ddd7a0274c9c64" Oct 30 01:06:13.129057 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.128976 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm"] Oct 30 01:06:13.133852 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.133820 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-ngzcm"] Oct 30 01:06:13.169735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.166383 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb"] Oct 30 01:06:13.248733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.247853 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-jj5wb"] Oct 30 01:06:13.563187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.563150 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj"] Oct 30 01:06:13.563882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.563866 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" Oct 30 01:06:13.563967 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.563911 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" Oct 30 01:06:13.563967 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.563934 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" Oct 30 01:06:13.563967 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.563940 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" Oct 30 01:06:13.564131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.564070 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" containerName="istio-proxy" Oct 30 01:06:13.564131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.564093 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" containerName="istio-proxy" Oct 30 01:06:13.568424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.568401 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.588106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.588076 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj"] Oct 30 01:06:13.682932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.682887 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d383e43-ztllj\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.683149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.683129 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcq4\" (UniqueName: \"kubernetes.io/projected/5edcfd09-2253-4132-87ef-d38e08356f36-kube-api-access-pbcq4\") pod \"maas-api-revoke-sub-4d383e43-ztllj\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.683212 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.683183 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d383e43-ztllj\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.785301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.785266 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d383e43-ztllj\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.785493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.785469 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcq4\" (UniqueName: \"kubernetes.io/projected/5edcfd09-2253-4132-87ef-d38e08356f36-kube-api-access-pbcq4\") pod \"maas-api-revoke-sub-4d383e43-ztllj\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.785587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.785520 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d383e43-ztllj\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.786157 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.786129 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d383e43-ztllj\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.791098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.791055 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d383e43-ztllj\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.808229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.808194 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcq4\" (UniqueName: \"kubernetes.io/projected/5edcfd09-2253-4132-87ef-d38e08356f36-kube-api-access-pbcq4\") pod \"maas-api-revoke-sub-4d383e43-ztllj\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:13.883911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:13.883826 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:14.201108 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:06:14.201074 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edcfd09_2253_4132_87ef_d38e08356f36.slice/crio-58686c65511e0036c344200f451dd6cff3805ef8e5189c9d18809ef73ca7fe71 WatchSource:0}: Error finding container 58686c65511e0036c344200f451dd6cff3805ef8e5189c9d18809ef73ca7fe71: Status 404 returned error can't find the container with id 58686c65511e0036c344200f451dd6cff3805ef8e5189c9d18809ef73ca7fe71 Oct 30 01:06:14.201750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.201680 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj"] Oct 30 01:06:14.210533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.210508 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:14.237802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.237766 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd" path="/var/lib/kubelet/pods/a0b676fb-dcc1-4e8d-bd05-5e33ac90d6dd/volumes" Oct 30 01:06:14.238540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.238522 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e883ea-a69a-4d3b-a80b-bc90b9edb040" path="/var/lib/kubelet/pods/a8e883ea-a69a-4d3b-a80b-bc90b9edb040/volumes" Oct 30 01:06:14.293965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.293917 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9efb88a8-9e61-444f-b996-32948d51920b-kube-api-access-mgltq\" (UniqueName: \"kubernetes.io/projected/9efb88a8-9e61-444f-b996-32948d51920b-kube-api-access-mgltq\") pod \"9efb88a8-9e61-444f-b996-32948d51920b\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " Oct 30 01:06:14.294156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.294107 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-openshift-ca\") pod \"9efb88a8-9e61-444f-b996-32948d51920b\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " Oct 30 01:06:14.294482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.294458 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-openshift-ca" pod "9efb88a8-9e61-444f-b996-32948d51920b" (UID: "9efb88a8-9e61-444f-b996-32948d51920b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:14.294779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.294761 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-xks-ca\") pod \"9efb88a8-9e61-444f-b996-32948d51920b\" (UID: \"9efb88a8-9e61-444f-b996-32948d51920b\") " Oct 30 01:06:14.295537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.295495 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:14.359609 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.359568 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-xks-ca" pod "9efb88a8-9e61-444f-b996-32948d51920b" (UID: "9efb88a8-9e61-444f-b996-32948d51920b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:14.359816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.359617 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efb88a8-9e61-444f-b996-32948d51920b-kube-api-access-mgltq" pod "9efb88a8-9e61-444f-b996-32948d51920b" (UID: "9efb88a8-9e61-444f-b996-32948d51920b"). InnerVolumeSpecName "kube-api-access-mgltq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:14.396958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.396925 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efb88a8-9e61-444f-b996-32948d51920b-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:14.396958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.396955 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgltq\" (UniqueName: \"kubernetes.io/projected/9efb88a8-9e61-444f-b996-32948d51920b-kube-api-access-mgltq\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:14.515491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:14.515360 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:15.058615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:15.058585 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" Oct 30 01:06:15.059349 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:15.059316 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2" event={"ID":"9efb88a8-9e61-444f-b996-32948d51920b","Type":"ContainerDied","Data":"e45952107f7f9403dbb287d8e0e2e61433d63e4402eb9a499b8599e516e82652"} Oct 30 01:06:15.059509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:15.059380 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e45952107f7f9403dbb287d8e0e2e61433d63e4402eb9a499b8599e516e82652" Oct 30 01:06:15.063150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:15.063119 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" event={"ID":"5edcfd09-2253-4132-87ef-d38e08356f36","Type":"ContainerStarted","Data":"3fcfb04f12cf3162f0b541f622ad6e13ae1d38f1bb1a9c521bb52e338a29bcff"} Oct 30 01:06:15.063290 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:15.063174 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" event={"ID":"5edcfd09-2253-4132-87ef-d38e08356f36","Type":"ContainerStarted","Data":"58686c65511e0036c344200f451dd6cff3805ef8e5189c9d18809ef73ca7fe71"} Oct 30 01:06:15.120231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:15.120175 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" podStartSLOduration=2.1201604 podStartE2EDuration="2.1201604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:15.114435399 +0000 UTC m=+1913.026562558" watchObservedRunningTime="2026-10-30 01:06:15.1201604 +0000 UTC m=+1913.032287540" Oct 30 01:06:16.071759 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:16.071697 4517 generic.go:356] "Generic (PLEG): container finished" podID="5edcfd09-2253-4132-87ef-d38e08356f36" containerID="3fcfb04f12cf3162f0b541f622ad6e13ae1d38f1bb1a9c521bb52e338a29bcff" exitCode=0 Oct 30 01:06:16.072185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:16.071779 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" event={"ID":"5edcfd09-2253-4132-87ef-d38e08356f36","Type":"ContainerDied","Data":"3fcfb04f12cf3162f0b541f622ad6e13ae1d38f1bb1a9c521bb52e338a29bcff"} Oct 30 01:06:16.180675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:16.180643 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:16.422062 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:16.421955 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:17.265136 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.265109 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:17.350291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.350191 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-openshift-ca\") pod \"5edcfd09-2253-4132-87ef-d38e08356f36\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " Oct 30 01:06:17.350291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.350270 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-xks-ca\") pod \"5edcfd09-2253-4132-87ef-d38e08356f36\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " Oct 30 01:06:17.350528 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.350313 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edcfd09-2253-4132-87ef-d38e08356f36-kube-api-access-pbcq4\" (UniqueName: \"kubernetes.io/projected/5edcfd09-2253-4132-87ef-d38e08356f36-kube-api-access-pbcq4\") pod \"5edcfd09-2253-4132-87ef-d38e08356f36\" (UID: \"5edcfd09-2253-4132-87ef-d38e08356f36\") " Oct 30 01:06:17.350655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.350626 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-openshift-ca" pod "5edcfd09-2253-4132-87ef-d38e08356f36" (UID: "5edcfd09-2253-4132-87ef-d38e08356f36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:17.351427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.351399 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:17.353128 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.353097 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-xks-ca" pod "5edcfd09-2253-4132-87ef-d38e08356f36" (UID: "5edcfd09-2253-4132-87ef-d38e08356f36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:17.353420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.353388 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edcfd09-2253-4132-87ef-d38e08356f36-kube-api-access-pbcq4" pod "5edcfd09-2253-4132-87ef-d38e08356f36" (UID: "5edcfd09-2253-4132-87ef-d38e08356f36"). InnerVolumeSpecName "kube-api-access-pbcq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:17.453200 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.453160 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edcfd09-2253-4132-87ef-d38e08356f36-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:17.453200 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:17.453195 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbcq4\" (UniqueName: \"kubernetes.io/projected/5edcfd09-2253-4132-87ef-d38e08356f36-kube-api-access-pbcq4\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:18.010665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:18.010616 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:18.111220 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:18.108550 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" event={"ID":"5edcfd09-2253-4132-87ef-d38e08356f36","Type":"ContainerDied","Data":"58686c65511e0036c344200f451dd6cff3805ef8e5189c9d18809ef73ca7fe71"} Oct 30 01:06:18.111220 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:18.108804 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj" Oct 30 01:06:18.111220 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:18.108760 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58686c65511e0036c344200f451dd6cff3805ef8e5189c9d18809ef73ca7fe71" Oct 30 01:06:19.582426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:19.582389 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:21.308300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:21.308266 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:22.886289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:22.886247 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:24.587865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:24.587829 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:24.657060 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:24.656995 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" podUID="2ba7987b-ebd3-4345-9a36-997cdc5b2754" containerName="main" containerID="cri-o://97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092" gracePeriod=45 Oct 30 01:06:24.842603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:24.842495 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" podUID="2ba7987b-ebd3-4345-9a36-997cdc5b2754" containerName="main" probeResult="failure" output="Get \"https://10.133.0.122:8000/ready\": dial tcp 10.133.0.122:8000: connect: connection refused" Oct 30 01:06:24.994520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:24.994495 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:06:25.093301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.093179 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-home\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-home\") pod \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " Oct 30 01:06:25.093761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.093725 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-home" pod "2ba7987b-ebd3-4345-9a36-997cdc5b2754" (UID: "2ba7987b-ebd3-4345-9a36-997cdc5b2754"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:25.093880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.093863 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tls-certs\") pod \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " Oct 30 01:06:25.093991 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.093975 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-dshm\") pod \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " Oct 30 01:06:25.094103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.094089 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba7987b-ebd3-4345-9a36-997cdc5b2754-kube-api-access-hn8j4\" (UniqueName: \"kubernetes.io/projected/2ba7987b-ebd3-4345-9a36-997cdc5b2754-kube-api-access-hn8j4\") pod \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " Oct 30 01:06:25.094292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.094276 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-model-cache\") pod \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " Oct 30 01:06:25.094350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.094317 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tmp-dir\") pod \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\" (UID: \"2ba7987b-ebd3-4345-9a36-997cdc5b2754\") " Oct 30 01:06:25.095213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.095185 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-model-cache" pod "2ba7987b-ebd3-4345-9a36-997cdc5b2754" (UID: "2ba7987b-ebd3-4345-9a36-997cdc5b2754"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:25.095302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.095212 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:25.095442 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.095416 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tmp-dir" pod "2ba7987b-ebd3-4345-9a36-997cdc5b2754" (UID: "2ba7987b-ebd3-4345-9a36-997cdc5b2754"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:25.097582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.097555 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba7987b-ebd3-4345-9a36-997cdc5b2754-kube-api-access-hn8j4" pod "2ba7987b-ebd3-4345-9a36-997cdc5b2754" (UID: "2ba7987b-ebd3-4345-9a36-997cdc5b2754"). InnerVolumeSpecName "kube-api-access-hn8j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:25.097582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.097567 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-dshm" pod "2ba7987b-ebd3-4345-9a36-997cdc5b2754" (UID: "2ba7987b-ebd3-4345-9a36-997cdc5b2754"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:25.098291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.098263 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tls-certs" pod "2ba7987b-ebd3-4345-9a36-997cdc5b2754" (UID: "2ba7987b-ebd3-4345-9a36-997cdc5b2754"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:25.165800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.165589 4517 generic.go:356] "Generic (PLEG): container finished" podID="2ba7987b-ebd3-4345-9a36-997cdc5b2754" containerID="97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092" exitCode=0 Oct 30 01:06:25.165800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.165630 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" event={"ID":"2ba7987b-ebd3-4345-9a36-997cdc5b2754","Type":"ContainerDied","Data":"97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092"} Oct 30 01:06:25.165800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.165675 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" Oct 30 01:06:25.165800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.165684 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx" event={"ID":"2ba7987b-ebd3-4345-9a36-997cdc5b2754","Type":"ContainerDied","Data":"9cd241fcd5a069287b20ab9a923eabc3385a5f1b6b2a5511c194c5dcf932cf1c"} Oct 30 01:06:25.165800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.165752 4517 scope.go:122] "RemoveContainer" containerID="97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092" Oct 30 01:06:25.180646 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.180616 4517 scope.go:122] "RemoveContainer" containerID="97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092" Oct 30 01:06:25.180984 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:06:25.180963 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092\": container with ID starting with 97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092 not found: ID does not exist" containerID="97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092" Oct 30 01:06:25.181049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.180994 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092"} err="failed to get container status \"97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092\": rpc error: code = NotFound desc = could not find container \"97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092\": container with ID starting with 97aac134f5e4d66671c870512fcd163e7360f613d71643e5eef3f5808aecf092 not found: ID does not exist" Oct 30 01:06:25.197751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.197727 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:25.197751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.197753 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:25.197911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.197768 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn8j4\" (UniqueName: \"kubernetes.io/projected/2ba7987b-ebd3-4345-9a36-997cdc5b2754-kube-api-access-hn8j4\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:25.197911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.197781 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:25.197911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.197795 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba7987b-ebd3-4345-9a36-997cdc5b2754-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:25.214941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.214914 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx"] Oct 30 01:06:25.223305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:25.223263 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5f4f9b/test-model-5f4f9b-kserve-69dfdc8b7f-jnlmx"] Oct 30 01:06:26.237593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:26.237554 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba7987b-ebd3-4345-9a36-997cdc5b2754" path="/var/lib/kubelet/pods/2ba7987b-ebd3-4345-9a36-997cdc5b2754/volumes" Oct 30 01:06:26.241944 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:26.241919 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:27.800600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:27.800571 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:29.569341 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:29.569307 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:30.762831 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:06:30.762795 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:06:31.200092 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.200050 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:31.513435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.513341 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:31.766274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.766190 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx"] Oct 30 01:06:31.767997 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.767970 4517 cpu_manager.go:414] "containerMap: removing container" podUID="5edcfd09-2253-4132-87ef-d38e08356f36" containerName="revoke-keys" Oct 30 01:06:31.770292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.770256 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edcfd09-2253-4132-87ef-d38e08356f36" containerName="revoke-keys" Oct 30 01:06:31.770486 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.770470 4517 cpu_manager.go:414] "containerMap: removing container" podUID="9efb88a8-9e61-444f-b996-32948d51920b" containerName="revoke-keys" Oct 30 01:06:31.776783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.776758 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9efb88a8-9e61-444f-b996-32948d51920b" containerName="revoke-keys" Oct 30 01:06:31.776952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.776928 4517 cpu_manager.go:414] "containerMap: removing container" podUID="2ba7987b-ebd3-4345-9a36-997cdc5b2754" containerName="main" Oct 30 01:06:31.777040 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.777028 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba7987b-ebd3-4345-9a36-997cdc5b2754" containerName="main" Oct 30 01:06:31.777383 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.777365 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edcfd09-2253-4132-87ef-d38e08356f36" containerName="revoke-keys" Oct 30 01:06:31.777538 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.777514 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="9efb88a8-9e61-444f-b996-32948d51920b" containerName="revoke-keys" Oct 30 01:06:31.777652 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.777639 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba7987b-ebd3-4345-9a36-997cdc5b2754" containerName="main" Oct 30 01:06:31.868602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.868568 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx"] Oct 30 01:06:31.868964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.868944 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:31.886147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.886118 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:06:31.896057 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.896035 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:06:31.952260 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.952222 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719e6d1f-1958-4a06-995c-19d788897524-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:31.952454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.952284 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719e6d1f-1958-4a06-995c-19d788897524-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:31.952454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:31.952438 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfwx\" (UniqueName: \"kubernetes.io/projected/719e6d1f-1958-4a06-995c-19d788897524-kube-api-access-wrfwx\") pod \"maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:32.054128 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:32.054077 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719e6d1f-1958-4a06-995c-19d788897524-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:32.054338 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:32.054162 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719e6d1f-1958-4a06-995c-19d788897524-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:32.054338 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:32.054259 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfwx\" (UniqueName: \"kubernetes.io/projected/719e6d1f-1958-4a06-995c-19d788897524-kube-api-access-wrfwx\") pod \"maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:32.055167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:32.055136 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719e6d1f-1958-4a06-995c-19d788897524-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:32.057008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:32.056980 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719e6d1f-1958-4a06-995c-19d788897524-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:32.075556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:32.075530 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfwx\" (UniqueName: \"kubernetes.io/projected/719e6d1f-1958-4a06-995c-19d788897524-kube-api-access-wrfwx\") pod \"maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:32.198578 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:32.198533 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:32.516111 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:32.516083 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx"] Oct 30 01:06:32.516377 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:06:32.516356 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719e6d1f_1958_4a06_995c_19d788897524.slice/crio-8dc3d82eec9cd8a121cae415f218f1d497954a546327997fa5d35338aa89e205 WatchSource:0}: Error finding container 8dc3d82eec9cd8a121cae415f218f1d497954a546327997fa5d35338aa89e205: Status 404 returned error can't find the container with id 8dc3d82eec9cd8a121cae415f218f1d497954a546327997fa5d35338aa89e205 Oct 30 01:06:33.108860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:33.108821 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:33.238169 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:33.238134 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" event={"ID":"719e6d1f-1958-4a06-995c-19d788897524","Type":"ContainerStarted","Data":"66e3648bd76c71390b7ee5a01c3224bbca08f32bab3f572e92210426a696fed3"} Oct 30 01:06:33.238335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:33.238189 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" event={"ID":"719e6d1f-1958-4a06-995c-19d788897524","Type":"ContainerStarted","Data":"8dc3d82eec9cd8a121cae415f218f1d497954a546327997fa5d35338aa89e205"} Oct 30 01:06:33.279522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:33.279444 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" podStartSLOduration=2.279423559 podStartE2EDuration="2.279423559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:33.274110095 +0000 UTC m=+1931.186237235" watchObservedRunningTime="2026-10-30 01:06:33.279423559 +0000 UTC m=+1931.191550701" Oct 30 01:06:34.247527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:34.247495 4517 generic.go:356] "Generic (PLEG): container finished" podID="719e6d1f-1958-4a06-995c-19d788897524" containerID="66e3648bd76c71390b7ee5a01c3224bbca08f32bab3f572e92210426a696fed3" exitCode=0 Oct 30 01:06:34.247948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:34.247572 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" event={"ID":"719e6d1f-1958-4a06-995c-19d788897524","Type":"ContainerDied","Data":"66e3648bd76c71390b7ee5a01c3224bbca08f32bab3f572e92210426a696fed3"} Oct 30 01:06:34.849738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:34.849198 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:34.977732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:34.977121 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:35.056614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.052609 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq"] Oct 30 01:06:35.064783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.064011 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.105783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.099798 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq"] Oct 30 01:06:35.192031 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.188677 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.192031 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.188829 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9hl\" (UniqueName: \"kubernetes.io/projected/7db4bde9-c203-419c-96fc-78b08ab29e4b-kube-api-access-9x9hl\") pod \"maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.192031 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.189089 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.290888 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.290846 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.291509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.291478 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.291638 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.291617 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9hl\" (UniqueName: \"kubernetes.io/projected/7db4bde9-c203-419c-96fc-78b08ab29e4b-kube-api-access-9x9hl\") pod \"maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.292040 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.292017 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.296664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.296620 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.313978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.313885 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9hl\" (UniqueName: \"kubernetes.io/projected/7db4bde9-c203-419c-96fc-78b08ab29e4b-kube-api-access-9x9hl\") pod \"maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.483528 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.483495 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:35.491683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.491656 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:35.600117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.598990 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719e6d1f-1958-4a06-995c-19d788897524-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719e6d1f-1958-4a06-995c-19d788897524-maas-api-openshift-ca\") pod \"719e6d1f-1958-4a06-995c-19d788897524\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " Oct 30 01:06:35.600117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.599216 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719e6d1f-1958-4a06-995c-19d788897524-kube-api-access-wrfwx\" (UniqueName: \"kubernetes.io/projected/719e6d1f-1958-4a06-995c-19d788897524-kube-api-access-wrfwx\") pod \"719e6d1f-1958-4a06-995c-19d788897524\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " Oct 30 01:06:35.600117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.599273 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719e6d1f-1958-4a06-995c-19d788897524-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719e6d1f-1958-4a06-995c-19d788897524-maas-api-xks-ca\") pod \"719e6d1f-1958-4a06-995c-19d788897524\" (UID: \"719e6d1f-1958-4a06-995c-19d788897524\") " Oct 30 01:06:35.600431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.600237 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719e6d1f-1958-4a06-995c-19d788897524-maas-api-openshift-ca" pod "719e6d1f-1958-4a06-995c-19d788897524" (UID: "719e6d1f-1958-4a06-995c-19d788897524"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:35.603016 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.602969 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719e6d1f-1958-4a06-995c-19d788897524-kube-api-access-wrfwx" pod "719e6d1f-1958-4a06-995c-19d788897524" (UID: "719e6d1f-1958-4a06-995c-19d788897524"). InnerVolumeSpecName "kube-api-access-wrfwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:35.603147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.603007 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719e6d1f-1958-4a06-995c-19d788897524-maas-api-xks-ca" pod "719e6d1f-1958-4a06-995c-19d788897524" (UID: "719e6d1f-1958-4a06-995c-19d788897524"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:35.701579 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.701483 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrfwx\" (UniqueName: \"kubernetes.io/projected/719e6d1f-1958-4a06-995c-19d788897524-kube-api-access-wrfwx\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:35.701579 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.701526 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719e6d1f-1958-4a06-995c-19d788897524-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:35.701579 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.701539 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719e6d1f-1958-4a06-995c-19d788897524-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:35.801878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:35.801036 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq"] Oct 30 01:06:35.807308 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:06:35.807263 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db4bde9_c203_419c_96fc_78b08ab29e4b.slice/crio-9467b450abcf736c9713a5ac1dd4d061b222c794c8d021cdf5ffac5fbf6ca967 WatchSource:0}: Error finding container 9467b450abcf736c9713a5ac1dd4d061b222c794c8d021cdf5ffac5fbf6ca967: Status 404 returned error can't find the container with id 9467b450abcf736c9713a5ac1dd4d061b222c794c8d021cdf5ffac5fbf6ca967 Oct 30 01:06:36.269899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.269856 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" event={"ID":"7db4bde9-c203-419c-96fc-78b08ab29e4b","Type":"ContainerStarted","Data":"64eabba66543d989acc35a5d362645f07c414f1208137b1f493d8e13955c14a1"} Oct 30 01:06:36.270093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.269928 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" event={"ID":"7db4bde9-c203-419c-96fc-78b08ab29e4b","Type":"ContainerStarted","Data":"9467b450abcf736c9713a5ac1dd4d061b222c794c8d021cdf5ffac5fbf6ca967"} Oct 30 01:06:36.271449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.271411 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" event={"ID":"719e6d1f-1958-4a06-995c-19d788897524","Type":"ContainerDied","Data":"8dc3d82eec9cd8a121cae415f218f1d497954a546327997fa5d35338aa89e205"} Oct 30 01:06:36.271587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.271456 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx" Oct 30 01:06:36.271587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.271461 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dc3d82eec9cd8a121cae415f218f1d497954a546327997fa5d35338aa89e205" Oct 30 01:06:36.309726 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.309665 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" podStartSLOduration=2.309648875 podStartE2EDuration="2.309648875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:36.305411844 +0000 UTC m=+1934.217538994" watchObservedRunningTime="2026-10-30 01:06:36.309648875 +0000 UTC m=+1934.221776014" Oct 30 01:06:36.405902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.405817 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4"] Oct 30 01:06:36.406130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.406097 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" containerID="cri-o://dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559" gracePeriod=30 Oct 30 01:06:36.447437 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.447395 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x"] Oct 30 01:06:36.447683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.447652 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" containerID="cri-o://d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706" gracePeriod=30 Oct 30 01:06:36.454036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.454010 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp"] Oct 30 01:06:36.454258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.454228 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" containerID="cri-o://b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f" gracePeriod=30 Oct 30 01:06:36.897438 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:36.897404 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:37.283866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:37.283762 4517 generic.go:356] "Generic (PLEG): container finished" podID="7db4bde9-c203-419c-96fc-78b08ab29e4b" containerID="64eabba66543d989acc35a5d362645f07c414f1208137b1f493d8e13955c14a1" exitCode=0 Oct 30 01:06:37.283866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:37.283811 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" event={"ID":"7db4bde9-c203-419c-96fc-78b08ab29e4b","Type":"ContainerDied","Data":"64eabba66543d989acc35a5d362645f07c414f1208137b1f493d8e13955c14a1"} Oct 30 01:06:38.497298 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.497271 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:38.535230 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.535183 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:38.551671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.551574 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db4bde9-c203-419c-96fc-78b08ab29e4b-kube-api-access-9x9hl\" (UniqueName: \"kubernetes.io/projected/7db4bde9-c203-419c-96fc-78b08ab29e4b-kube-api-access-9x9hl\") pod \"7db4bde9-c203-419c-96fc-78b08ab29e4b\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " Oct 30 01:06:38.551671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.551646 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-xks-ca\") pod \"7db4bde9-c203-419c-96fc-78b08ab29e4b\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " Oct 30 01:06:38.551977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.551840 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-openshift-ca\") pod \"7db4bde9-c203-419c-96fc-78b08ab29e4b\" (UID: \"7db4bde9-c203-419c-96fc-78b08ab29e4b\") " Oct 30 01:06:38.552458 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.552431 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-openshift-ca" pod "7db4bde9-c203-419c-96fc-78b08ab29e4b" (UID: "7db4bde9-c203-419c-96fc-78b08ab29e4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:38.552858 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.552835 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:38.554397 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.554368 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-xks-ca" pod "7db4bde9-c203-419c-96fc-78b08ab29e4b" (UID: "7db4bde9-c203-419c-96fc-78b08ab29e4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:38.555219 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.555184 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db4bde9-c203-419c-96fc-78b08ab29e4b-kube-api-access-9x9hl" pod "7db4bde9-c203-419c-96fc-78b08ab29e4b" (UID: "7db4bde9-c203-419c-96fc-78b08ab29e4b"). InnerVolumeSpecName "kube-api-access-9x9hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:38.654192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.654158 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x9hl\" (UniqueName: \"kubernetes.io/projected/7db4bde9-c203-419c-96fc-78b08ab29e4b-kube-api-access-9x9hl\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:38.654192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:38.654191 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db4bde9-c203-419c-96fc-78b08ab29e4b-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:39.305733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:39.305008 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" Oct 30 01:06:39.305733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:39.305029 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq" event={"ID":"7db4bde9-c203-419c-96fc-78b08ab29e4b","Type":"ContainerDied","Data":"9467b450abcf736c9713a5ac1dd4d061b222c794c8d021cdf5ffac5fbf6ca967"} Oct 30 01:06:39.305733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:39.305090 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9467b450abcf736c9713a5ac1dd4d061b222c794c8d021cdf5ffac5fbf6ca967" Oct 30 01:06:40.171606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:40.171577 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:41.795644 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.795617 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 01:06:41.848853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.848828 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 01:06:41.896359 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.896325 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 01:06:41.910873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.910734 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-kube-api-access-hr64w\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-kube-api-access-hr64w\") pod \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " Oct 30 01:06:41.910873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.910803 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-data\") pod \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " Oct 30 01:06:41.910873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.910863 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-certs\") pod \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.910993 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-socket\") pod \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.911078 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-certs\") pod \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.911135 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-podinfo\") pod \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.911162 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-socket\") pod \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.911225 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-podinfo\") pod \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.911306 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-data\") pod \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.911365 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-envoy\") pod \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.911402 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-token\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-token\") pod \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.912268 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-envoy\") pod \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.912352 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-token\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-token\") pod \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " Oct 30 01:06:41.912506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.912507 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istiod-ca-cert\") pod \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " Oct 30 01:06:41.913131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.912552 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-credential-socket\") pod \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " Oct 30 01:06:41.913586 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.913323 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istiod-ca-cert\") pod \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\" (UID: \"ac7a6065-cd08-45df-b7ae-b3f42590dca7\") " Oct 30 01:06:41.915367 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.913425 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-credential-socket\") pod \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " Oct 30 01:06:41.915367 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.914884 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-kube-api-access-4b9pj\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-kube-api-access-4b9pj\") pod \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\" (UID: \"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4\") " Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.919281 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-podinfo" pod "ac7a6065-cd08-45df-b7ae-b3f42590dca7" (UID: "ac7a6065-cd08-45df-b7ae-b3f42590dca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.919370 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-socket" pod "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" (UID: "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.919403 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-data" pod "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" (UID: "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.919690 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-socket" pod "ac7a6065-cd08-45df-b7ae-b3f42590dca7" (UID: "ac7a6065-cd08-45df-b7ae-b3f42590dca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.920283 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-certs" pod "ac7a6065-cd08-45df-b7ae-b3f42590dca7" (UID: "ac7a6065-cd08-45df-b7ae-b3f42590dca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.920484 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-credential-socket" pod "ac7a6065-cd08-45df-b7ae-b3f42590dca7" (UID: "ac7a6065-cd08-45df-b7ae-b3f42590dca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.920789 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istiod-ca-cert" pod "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" (UID: "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.920842 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-kube-api-access-4b9pj" pod "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" (UID: "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4"). InnerVolumeSpecName "kube-api-access-4b9pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.921060 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-credential-socket" pod "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" (UID: "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.921139 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istiod-ca-cert" pod "ac7a6065-cd08-45df-b7ae-b3f42590dca7" (UID: "ac7a6065-cd08-45df-b7ae-b3f42590dca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:41.923737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.922333 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-token" pod "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" (UID: "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:41.924429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.923913 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-envoy" pod "ac7a6065-cd08-45df-b7ae-b3f42590dca7" (UID: "ac7a6065-cd08-45df-b7ae-b3f42590dca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.924429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.924106 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-data" pod "ac7a6065-cd08-45df-b7ae-b3f42590dca7" (UID: "ac7a6065-cd08-45df-b7ae-b3f42590dca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.925761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.924965 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-certs" pod "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" (UID: "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.925761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.925093 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-token" pod "ac7a6065-cd08-45df-b7ae-b3f42590dca7" (UID: "ac7a6065-cd08-45df-b7ae-b3f42590dca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:41.926087 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.926065 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-envoy" pod "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" (UID: "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:41.930078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.930043 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-podinfo" pod "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" (UID: "738a79a8-d5e3-4a95-9d05-ec6b3beb66f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:41.931783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.931748 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-kube-api-access-hr64w" pod "ac7a6065-cd08-45df-b7ae-b3f42590dca7" (UID: "ac7a6065-cd08-45df-b7ae-b3f42590dca7"). InnerVolumeSpecName "kube-api-access-hr64w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:41.935849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:41.935827 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:42.016642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.016540 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-token\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-token\") pod \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " Oct 30 01:06:42.016642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.016600 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-envoy\") pod \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " Oct 30 01:06:42.016973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.016729 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-podinfo\") pod \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " Oct 30 01:06:42.016973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.016846 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-data\") pod \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " Oct 30 01:06:42.016973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.016877 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-certs\") pod \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " Oct 30 01:06:42.016973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.016944 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-socket\") pod \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " Oct 30 01:06:42.017296 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.017010 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-credential-socket\") pod \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " Oct 30 01:06:42.017296 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.017191 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-kube-api-access-qkv6l\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-kube-api-access-qkv6l\") pod \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " Oct 30 01:06:42.017296 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.017248 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istiod-ca-cert\") pod \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\" (UID: \"701f808d-ca6b-4e78-aa66-2516a24f0ed7\") " Oct 30 01:06:42.017515 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.017381 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-data" pod "701f808d-ca6b-4e78-aa66-2516a24f0ed7" (UID: "701f808d-ca6b-4e78-aa66-2516a24f0ed7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:42.018011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.017985 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018013 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018026 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018038 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018051 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018063 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018074 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018086 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018109 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7a6065-cd08-45df-b7ae-b3f42590dca7-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018122 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018134 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018146 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b9pj\" (UniqueName: \"kubernetes.io/projected/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-kube-api-access-4b9pj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018160 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr64w\" (UniqueName: \"kubernetes.io/projected/ac7a6065-cd08-45df-b7ae-b3f42590dca7-kube-api-access-hr64w\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018173 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018185 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018197 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7a6065-cd08-45df-b7ae-b3f42590dca7-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018209 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018222 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018233 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018299 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-socket" pod "701f808d-ca6b-4e78-aa66-2516a24f0ed7" (UID: "701f808d-ca6b-4e78-aa66-2516a24f0ed7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018488 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-certs" pod "701f808d-ca6b-4e78-aa66-2516a24f0ed7" (UID: "701f808d-ca6b-4e78-aa66-2516a24f0ed7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018543 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istiod-ca-cert" pod "701f808d-ca6b-4e78-aa66-2516a24f0ed7" (UID: "701f808d-ca6b-4e78-aa66-2516a24f0ed7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:42.018937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.018678 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-credential-socket" pod "701f808d-ca6b-4e78-aa66-2516a24f0ed7" (UID: "701f808d-ca6b-4e78-aa66-2516a24f0ed7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:42.021958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.021918 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-envoy" pod "701f808d-ca6b-4e78-aa66-2516a24f0ed7" (UID: "701f808d-ca6b-4e78-aa66-2516a24f0ed7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:42.027817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.027692 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-token" pod "701f808d-ca6b-4e78-aa66-2516a24f0ed7" (UID: "701f808d-ca6b-4e78-aa66-2516a24f0ed7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:42.028083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.027695 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-podinfo" pod "701f808d-ca6b-4e78-aa66-2516a24f0ed7" (UID: "701f808d-ca6b-4e78-aa66-2516a24f0ed7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:42.029278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.029258 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-kube-api-access-qkv6l" pod "701f808d-ca6b-4e78-aa66-2516a24f0ed7" (UID: "701f808d-ca6b-4e78-aa66-2516a24f0ed7"). InnerVolumeSpecName "kube-api-access-qkv6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:42.131734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.122308 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.131734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.122357 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.131734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.122369 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.131734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.122385 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkv6l\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-kube-api-access-qkv6l\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.131734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.122401 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.131734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.122413 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.131734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.122425 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.131734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.122437 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701f808d-ca6b-4e78-aa66-2516a24f0ed7-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.332206 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.332021 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:42.336173 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.336140 4517 generic.go:356] "Generic (PLEG): container finished" podID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerID="dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559" exitCode=0 Oct 30 01:06:42.336324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.336207 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" Oct 30 01:06:42.336324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.336256 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" event={"ID":"ac7a6065-cd08-45df-b7ae-b3f42590dca7","Type":"ContainerDied","Data":"dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559"} Oct 30 01:06:42.336324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.336304 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4" event={"ID":"ac7a6065-cd08-45df-b7ae-b3f42590dca7","Type":"ContainerDied","Data":"f76cb5efc8168c6a27304e5fa6e0bd36af4965b614b5921641eaba3ee4b18e63"} Oct 30 01:06:42.336517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.336332 4517 scope.go:122] "RemoveContainer" containerID="dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559" Oct 30 01:06:42.338958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.338932 4517 generic.go:356] "Generic (PLEG): container finished" podID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerID="b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f" exitCode=0 Oct 30 01:06:42.339105 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.339074 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" event={"ID":"701f808d-ca6b-4e78-aa66-2516a24f0ed7","Type":"ContainerDied","Data":"b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f"} Oct 30 01:06:42.339172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.339115 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" event={"ID":"701f808d-ca6b-4e78-aa66-2516a24f0ed7","Type":"ContainerDied","Data":"9f9d282313e17da6f3ff8fe4a991dfd6993093ee85e7feb8cbaf9caee25dc4a9"} Oct 30 01:06:42.339232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.339183 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp" Oct 30 01:06:42.341566 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.341542 4517 generic.go:356] "Generic (PLEG): container finished" podID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerID="d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706" exitCode=0 Oct 30 01:06:42.341661 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.341643 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" event={"ID":"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4","Type":"ContainerDied","Data":"d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706"} Oct 30 01:06:42.341785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.341679 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" event={"ID":"738a79a8-d5e3-4a95-9d05-ec6b3beb66f4","Type":"ContainerDied","Data":"2a692e3349e5345c23512a4cf0d9e1956b2dc983aee7f4e80c2f7bfd2f079f9e"} Oct 30 01:06:42.341785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.341766 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x" Oct 30 01:06:42.352188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.352169 4517 scope.go:122] "RemoveContainer" containerID="dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559" Oct 30 01:06:42.352455 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:06:42.352437 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559\": container with ID starting with dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559 not found: ID does not exist" containerID="dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559" Oct 30 01:06:42.352498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.352463 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559"} err="failed to get container status \"dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559\": rpc error: code = NotFound desc = could not find container \"dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559\": container with ID starting with dcc8e4c821db3ca83e855d745b840cf7b698d498ea963694b3d9d0b4d551a559 not found: ID does not exist" Oct 30 01:06:42.352498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.352478 4517 scope.go:122] "RemoveContainer" containerID="b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f" Oct 30 01:06:42.362227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.362209 4517 scope.go:122] "RemoveContainer" containerID="b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f" Oct 30 01:06:42.362518 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:06:42.362498 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f\": container with ID starting with b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f not found: ID does not exist" containerID="b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f" Oct 30 01:06:42.362592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.362524 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f"} err="failed to get container status \"b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f\": rpc error: code = NotFound desc = could not find container \"b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f\": container with ID starting with b151cdf943e6f6791ab1ccc3a54eb19a8e826119d4f56b01b6908b865417637f not found: ID does not exist" Oct 30 01:06:42.362592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.362547 4517 scope.go:122] "RemoveContainer" containerID="d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706" Oct 30 01:06:42.372317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.372292 4517 scope.go:122] "RemoveContainer" containerID="d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706" Oct 30 01:06:42.372618 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:06:42.372599 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706\": container with ID starting with d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706 not found: ID does not exist" containerID="d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706" Oct 30 01:06:42.372698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.372630 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706"} err="failed to get container status \"d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706\": rpc error: code = NotFound desc = could not find container \"d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706\": container with ID starting with d97a6dc89311abd7f01135a0e5823102447240e3c42a390640ee5c21fb45e706 not found: ID does not exist" Oct 30 01:06:42.388435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.388408 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4"] Oct 30 01:06:42.395171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.395148 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-b8rm4"] Oct 30 01:06:42.430681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.430638 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp"] Oct 30 01:06:42.453368 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.452540 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:42.457102 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.457017 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-vx4jp"] Oct 30 01:06:42.475192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.475160 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x"] Oct 30 01:06:42.486928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:42.486878 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-gml7x"] Oct 30 01:06:43.696660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:43.696620 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:44.239268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:44.237044 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" path="/var/lib/kubelet/pods/701f808d-ca6b-4e78-aa66-2516a24f0ed7/volumes" Oct 30 01:06:44.239268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:44.237783 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" path="/var/lib/kubelet/pods/738a79a8-d5e3-4a95-9d05-ec6b3beb66f4/volumes" Oct 30 01:06:44.239268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:44.238209 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" path="/var/lib/kubelet/pods/ac7a6065-cd08-45df-b7ae-b3f42590dca7/volumes" Oct 30 01:06:45.401316 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:45.401285 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:45.652492 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:45.652417 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:45.768520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:45.768486 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:47.405752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:47.405688 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:49.764733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:49.761885 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:49.803733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:49.784208 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:49.812735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:49.805781 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:49.812735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:49.810521 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:49.812735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:49.811405 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 01:06:49.836342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:49.836295 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:49.943734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:49.923825 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:49.986737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:49.986076 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:49.986737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:49.986463 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 01:06:50.043735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.041810 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" podUID="a7049336-fa54-49d4-9d60-4bd14f9b5ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.175735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.175414 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": context deadline exceeded" Oct 30 01:06:50.205732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.204933 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:50.525797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.516125 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.809734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.800788 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.809734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.801848 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:06:50.816736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.810841 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.816736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.811141 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 01:06:50.841787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.837917 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:06:50.841787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.838407 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 01:06:50.944735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.932827 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 01:06:50.944735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:50.933411 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 01:06:51.040734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:51.019518 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 01:06:51.040734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:51.032737 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:06:51.360551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:51.348378 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 01:06:51.431354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:51.431151 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 01:06:52.067141 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:52.067103 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:53.477876 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:53.477838 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.204784 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:54.204746 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.395826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:54.395796 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.462290 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:54.462204 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx"] Oct 30 01:06:54.466464 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:54.466438 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5f4f9b-jfdjx"] Oct 30 01:06:54.530687 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:54.530655 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.946671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:54.946636 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:56.237380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:56.237346 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719e6d1f-1958-4a06-995c-19d788897524" path="/var/lib/kubelet/pods/719e6d1f-1958-4a06-995c-19d788897524/volumes" Oct 30 01:06:56.439533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:56.439492 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:58.150638 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:58.150602 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:59.757646 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:06:59.757607 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:00.761475 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:07:00.761434 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:07:01.262372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:01.262339 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:01.380073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:01.380032 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:01.535681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:01.535645 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:01.587181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:01.587152 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h"] Oct 30 01:07:01.587400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:01.587376 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" containerName="istio-proxy" containerID="cri-o://963b00ccc8716f1cf0fc558095ad81b10c97e87f76b621dc441898875f83fd0d" gracePeriod=30 Oct 30 01:07:01.691734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:01.690233 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:02.104986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.104953 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg"] Oct 30 01:07:02.105846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105825 4517 cpu_manager.go:414] "containerMap: removing container" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" Oct 30 01:07:02.105846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105847 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" Oct 30 01:07:02.106030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105884 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" Oct 30 01:07:02.106030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105893 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" Oct 30 01:07:02.106030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105905 4517 cpu_manager.go:414] "containerMap: removing container" podUID="719e6d1f-1958-4a06-995c-19d788897524" containerName="revoke-keys" Oct 30 01:07:02.106030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105912 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719e6d1f-1958-4a06-995c-19d788897524" containerName="revoke-keys" Oct 30 01:07:02.106030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105937 4517 cpu_manager.go:414] "containerMap: removing container" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" Oct 30 01:07:02.106030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105945 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" Oct 30 01:07:02.106030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105959 4517 cpu_manager.go:414] "containerMap: removing container" podUID="7db4bde9-c203-419c-96fc-78b08ab29e4b" containerName="revoke-keys" Oct 30 01:07:02.106030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.105965 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db4bde9-c203-419c-96fc-78b08ab29e4b" containerName="revoke-keys" Oct 30 01:07:02.106428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.106158 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="738a79a8-d5e3-4a95-9d05-ec6b3beb66f4" containerName="istio-proxy" Oct 30 01:07:02.106428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.106173 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db4bde9-c203-419c-96fc-78b08ab29e4b" containerName="revoke-keys" Oct 30 01:07:02.106428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.106191 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="701f808d-ca6b-4e78-aa66-2516a24f0ed7" containerName="istio-proxy" Oct 30 01:07:02.106428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.106207 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7a6065-cd08-45df-b7ae-b3f42590dca7" containerName="istio-proxy" Oct 30 01:07:02.106428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.106215 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="719e6d1f-1958-4a06-995c-19d788897524" containerName="revoke-keys" Oct 30 01:07:02.111227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.111206 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.128817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.128793 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg"] Oct 30 01:07:02.152028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.151913 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:07:02.152028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.151995 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:07:02.296638 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.296598 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwv8n\" (UniqueName: \"kubernetes.io/projected/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-kube-api-access-bwv8n\") pod \"maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.296856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.296668 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.296856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.296811 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.398620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.398522 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwv8n\" (UniqueName: \"kubernetes.io/projected/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-kube-api-access-bwv8n\") pod \"maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.398845 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.398654 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.398845 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.398788 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.399953 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.399924 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.403187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.403156 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.425250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.425224 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwv8n\" (UniqueName: \"kubernetes.io/projected/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-kube-api-access-bwv8n\") pod \"maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.425444 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.425432 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:02.753113 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.753086 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg"] Oct 30 01:07:02.754816 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:07:02.754789 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104d83cc_f44d_40c3_aaf6_f7e91c2f498c.slice/crio-5062c0003de27e031483125988e49d3f8de617d35bba3149eb08d501e5f7375f WatchSource:0}: Error finding container 5062c0003de27e031483125988e49d3f8de617d35bba3149eb08d501e5f7375f: Status 404 returned error can't find the container with id 5062c0003de27e031483125988e49d3f8de617d35bba3149eb08d501e5f7375f Oct 30 01:07:02.811004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:02.810966 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" event={"ID":"104d83cc-f44d-40c3-aaf6-f7e91c2f498c","Type":"ContainerStarted","Data":"5062c0003de27e031483125988e49d3f8de617d35bba3149eb08d501e5f7375f"} Oct 30 01:07:03.450241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:03.450151 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:03.821508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:03.821474 4517 generic.go:356] "Generic (PLEG): container finished" podID="104d83cc-f44d-40c3-aaf6-f7e91c2f498c" containerID="64f017377cd1ea9b078b7190a67f11f3d9dd0e9e8ab4a4e28aa03feddb96834f" exitCode=0 Oct 30 01:07:03.821680 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:03.821526 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" event={"ID":"104d83cc-f44d-40c3-aaf6-f7e91c2f498c","Type":"ContainerDied","Data":"64f017377cd1ea9b078b7190a67f11f3d9dd0e9e8ab4a4e28aa03feddb96834f"} Oct 30 01:07:05.024474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.024447 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:05.046113 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.046087 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-kube-api-access-bwv8n\" (UniqueName: \"kubernetes.io/projected/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-kube-api-access-bwv8n\") pod \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " Oct 30 01:07:05.046307 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.046294 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-openshift-ca\") pod \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " Oct 30 01:07:05.046355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.046328 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-xks-ca\") pod \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\" (UID: \"104d83cc-f44d-40c3-aaf6-f7e91c2f498c\") " Oct 30 01:07:05.046696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.046668 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-openshift-ca" pod "104d83cc-f44d-40c3-aaf6-f7e91c2f498c" (UID: "104d83cc-f44d-40c3-aaf6-f7e91c2f498c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:05.046846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.046832 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:05.048336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.048309 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-kube-api-access-bwv8n" pod "104d83cc-f44d-40c3-aaf6-f7e91c2f498c" (UID: "104d83cc-f44d-40c3-aaf6-f7e91c2f498c"). InnerVolumeSpecName "kube-api-access-bwv8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:05.048506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.048488 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-xks-ca" pod "104d83cc-f44d-40c3-aaf6-f7e91c2f498c" (UID: "104d83cc-f44d-40c3-aaf6-f7e91c2f498c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:07:05.148685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.148608 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwv8n\" (UniqueName: \"kubernetes.io/projected/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-kube-api-access-bwv8n\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:05.148685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.148639 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104d83cc-f44d-40c3-aaf6-f7e91c2f498c-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:05.323322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.323290 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:05.838620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.838590 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" Oct 30 01:07:05.838835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.838621 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg" event={"ID":"104d83cc-f44d-40c3-aaf6-f7e91c2f498c","Type":"ContainerDied","Data":"5062c0003de27e031483125988e49d3f8de617d35bba3149eb08d501e5f7375f"} Oct 30 01:07:05.838835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:05.838679 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5062c0003de27e031483125988e49d3f8de617d35bba3149eb08d501e5f7375f" Oct 30 01:07:06.851864 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.851822 4517 generic.go:356] "Generic (PLEG): container finished" podID="65166d57-ef62-4bab-8c93-3441539c3c21" containerID="963b00ccc8716f1cf0fc558095ad81b10c97e87f76b621dc441898875f83fd0d" exitCode=0 Oct 30 01:07:06.852394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.852125 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" event={"ID":"65166d57-ef62-4bab-8c93-3441539c3c21","Type":"ContainerDied","Data":"963b00ccc8716f1cf0fc558095ad81b10c97e87f76b621dc441898875f83fd0d"} Oct 30 01:07:06.940399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.940361 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:07:06.977484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.977451 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-envoy\") pod \"65166d57-ef62-4bab-8c93-3441539c3c21\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " Oct 30 01:07:06.977683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.977602 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-certs\") pod \"65166d57-ef62-4bab-8c93-3441539c3c21\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " Oct 30 01:07:06.977683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.977638 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65166d57-ef62-4bab-8c93-3441539c3c21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65166d57-ef62-4bab-8c93-3441539c3c21-istiod-ca-cert\") pod \"65166d57-ef62-4bab-8c93-3441539c3c21\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " Oct 30 01:07:06.979068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.978263 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65166d57-ef62-4bab-8c93-3441539c3c21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65166d57-ef62-4bab-8c93-3441539c3c21-istio-podinfo\") pod \"65166d57-ef62-4bab-8c93-3441539c3c21\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " Oct 30 01:07:06.979068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.978312 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-credential-socket\") pod \"65166d57-ef62-4bab-8c93-3441539c3c21\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " Oct 30 01:07:06.979068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.978433 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-kube-api-access-nqdlb\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-kube-api-access-nqdlb\") pod \"65166d57-ef62-4bab-8c93-3441539c3c21\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " Oct 30 01:07:06.979068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.978500 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-socket\") pod \"65166d57-ef62-4bab-8c93-3441539c3c21\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " Oct 30 01:07:06.979068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.978528 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-data\") pod \"65166d57-ef62-4bab-8c93-3441539c3c21\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " Oct 30 01:07:06.979068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.978610 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-istio-token\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-istio-token\") pod \"65166d57-ef62-4bab-8c93-3441539c3c21\" (UID: \"65166d57-ef62-4bab-8c93-3441539c3c21\") " Oct 30 01:07:06.980794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.980652 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-socket" pod "65166d57-ef62-4bab-8c93-3441539c3c21" (UID: "65166d57-ef62-4bab-8c93-3441539c3c21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:06.981105 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.981067 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-data" pod "65166d57-ef62-4bab-8c93-3441539c3c21" (UID: "65166d57-ef62-4bab-8c93-3441539c3c21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:06.981396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.981300 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-certs" pod "65166d57-ef62-4bab-8c93-3441539c3c21" (UID: "65166d57-ef62-4bab-8c93-3441539c3c21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:06.981396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.981370 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-credential-socket" pod "65166d57-ef62-4bab-8c93-3441539c3c21" (UID: "65166d57-ef62-4bab-8c93-3441539c3c21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:06.981893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.981855 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65166d57-ef62-4bab-8c93-3441539c3c21-istiod-ca-cert" pod "65166d57-ef62-4bab-8c93-3441539c3c21" (UID: "65166d57-ef62-4bab-8c93-3441539c3c21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:06.983770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.983692 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-kube-api-access-nqdlb" pod "65166d57-ef62-4bab-8c93-3441539c3c21" (UID: "65166d57-ef62-4bab-8c93-3441539c3c21"). InnerVolumeSpecName "kube-api-access-nqdlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:06.985212 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.985181 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-envoy" pod "65166d57-ef62-4bab-8c93-3441539c3c21" (UID: "65166d57-ef62-4bab-8c93-3441539c3c21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:07.001369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:06.987999 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-istio-token" pod "65166d57-ef62-4bab-8c93-3441539c3c21" (UID: "65166d57-ef62-4bab-8c93-3441539c3c21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:07.006301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.005541 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65166d57-ef62-4bab-8c93-3441539c3c21-istio-podinfo" pod "65166d57-ef62-4bab-8c93-3441539c3c21" (UID: "65166d57-ef62-4bab-8c93-3441539c3c21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:07.017096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.017055 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:07.100733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.093486 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.100733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.093523 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.100733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.093547 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.100733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.093565 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.100733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.093584 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.100733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.093602 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65166d57-ef62-4bab-8c93-3441539c3c21-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.100733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.093620 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65166d57-ef62-4bab-8c93-3441539c3c21-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.100733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.093639 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65166d57-ef62-4bab-8c93-3441539c3c21-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.100733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.093661 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqdlb\" (UniqueName: \"kubernetes.io/projected/65166d57-ef62-4bab-8c93-3441539c3c21-kube-api-access-nqdlb\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.860816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.860769 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" event={"ID":"65166d57-ef62-4bab-8c93-3441539c3c21","Type":"ContainerDied","Data":"1eddbf5d8f1c333dd619892c50802aaeb163e6cdae964d27628a11bcb10066c9"} Oct 30 01:07:07.861278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.860845 4517 scope.go:122] "RemoveContainer" containerID="963b00ccc8716f1cf0fc558095ad81b10c97e87f76b621dc441898875f83fd0d" Oct 30 01:07:07.861278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.860780 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h" Oct 30 01:07:07.920614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.920569 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h"] Oct 30 01:07:07.930813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:07.930780 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-fhw7h"] Oct 30 01:07:08.237430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:08.237350 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" path="/var/lib/kubelet/pods/65166d57-ef62-4bab-8c93-3441539c3c21/volumes" Oct 30 01:07:08.541735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:08.541679 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:09.121740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:09.117245 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:09.324090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:09.324058 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:09.362437 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:09.362407 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq"] Oct 30 01:07:09.369026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:09.368987 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43c02d-c7rvq"] Oct 30 01:07:09.444239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:09.444150 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:09.855541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:09.855504 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:10.237255 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:10.237175 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db4bde9-c203-419c-96fc-78b08ab29e4b" path="/var/lib/kubelet/pods/7db4bde9-c203-419c-96fc-78b08ab29e4b/volumes" Oct 30 01:07:11.268962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:11.268927 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:12.685972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:12.685915 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:12.810109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:12.810045 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:13.140794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:13.140763 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:14.903981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:14.903943 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:16.400137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:16.400105 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:16.801152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:16.801122 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:16.923055 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:16.923017 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:17.138356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.138224 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:17.283265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.283214 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:17.666995 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.666953 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr"] Oct 30 01:07:17.667728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.667696 4517 cpu_manager.go:414] "containerMap: removing container" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" containerName="istio-proxy" Oct 30 01:07:17.667798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.667729 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" containerName="istio-proxy" Oct 30 01:07:17.667798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.667748 4517 cpu_manager.go:414] "containerMap: removing container" podUID="104d83cc-f44d-40c3-aaf6-f7e91c2f498c" containerName="revoke-keys" Oct 30 01:07:17.667798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.667754 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104d83cc-f44d-40c3-aaf6-f7e91c2f498c" containerName="revoke-keys" Oct 30 01:07:17.667916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.667880 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="104d83cc-f44d-40c3-aaf6-f7e91c2f498c" containerName="revoke-keys" Oct 30 01:07:17.667916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.667911 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="65166d57-ef62-4bab-8c93-3441539c3c21" containerName="istio-proxy" Oct 30 01:07:17.671777 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.671755 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.694669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.694636 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr"] Oct 30 01:07:17.716981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.716945 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:07:17.717185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.717022 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:07:17.802733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.802670 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.802940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.802738 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.802940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.802904 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drgt\" (UniqueName: \"kubernetes.io/projected/ab07e89d-1bba-4db9-97e2-e84dd54ca160-kube-api-access-9drgt\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.904221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.904184 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.904221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.904223 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.904482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.904279 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9drgt\" (UniqueName: \"kubernetes.io/projected/ab07e89d-1bba-4db9-97e2-e84dd54ca160-kube-api-access-9drgt\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.905134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.905051 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.906975 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.906953 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.926354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.926269 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drgt\" (UniqueName: \"kubernetes.io/projected/ab07e89d-1bba-4db9-97e2-e84dd54ca160-kube-api-access-9drgt\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:17.995670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:17.995628 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:18.318342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:18.318303 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr"] Oct 30 01:07:18.319744 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:07:18.319688 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab07e89d_1bba_4db9_97e2_e84dd54ca160.slice/crio-b3e896ce582099d7758c4199f4e47539a3f738086d875ee790f8b61bf663d74f WatchSource:0}: Error finding container b3e896ce582099d7758c4199f4e47539a3f738086d875ee790f8b61bf663d74f: Status 404 returned error can't find the container with id b3e896ce582099d7758c4199f4e47539a3f738086d875ee790f8b61bf663d74f Oct 30 01:07:19.008618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:19.008532 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" event={"ID":"ab07e89d-1bba-4db9-97e2-e84dd54ca160","Type":"ContainerStarted","Data":"918ceb1abde4bdd3c4a291176cad28c10067985c68e6b307765e65c5ca2e8888"} Oct 30 01:07:19.008618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:19.008589 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" event={"ID":"ab07e89d-1bba-4db9-97e2-e84dd54ca160","Type":"ContainerStarted","Data":"b3e896ce582099d7758c4199f4e47539a3f738086d875ee790f8b61bf663d74f"} Oct 30 01:07:19.049860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:19.049799 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" podStartSLOduration=2.04978179 podStartE2EDuration="2.04978179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:19.044493355 +0000 UTC m=+1976.956620498" watchObservedRunningTime="2026-10-30 01:07:19.04978179 +0000 UTC m=+1976.961908931" Oct 30 01:07:19.679560 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:19.679474 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:20.018559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:20.018464 4517 generic.go:356] "Generic (PLEG): container finished" podID="ab07e89d-1bba-4db9-97e2-e84dd54ca160" containerID="918ceb1abde4bdd3c4a291176cad28c10067985c68e6b307765e65c5ca2e8888" exitCode=0 Oct 30 01:07:20.018559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:20.018551 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" event={"ID":"ab07e89d-1bba-4db9-97e2-e84dd54ca160","Type":"ContainerDied","Data":"918ceb1abde4bdd3c4a291176cad28c10067985c68e6b307765e65c5ca2e8888"} Oct 30 01:07:21.201268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.201240 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:21.253614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.253582 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-xks-ca\") pod \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " Oct 30 01:07:21.253831 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.253664 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-openshift-ca\") pod \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " Oct 30 01:07:21.253831 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.253774 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab07e89d-1bba-4db9-97e2-e84dd54ca160-kube-api-access-9drgt\" (UniqueName: \"kubernetes.io/projected/ab07e89d-1bba-4db9-97e2-e84dd54ca160-kube-api-access-9drgt\") pod \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\" (UID: \"ab07e89d-1bba-4db9-97e2-e84dd54ca160\") " Oct 30 01:07:21.254101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.254074 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-openshift-ca" pod "ab07e89d-1bba-4db9-97e2-e84dd54ca160" (UID: "ab07e89d-1bba-4db9-97e2-e84dd54ca160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:21.254675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.254649 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:21.256017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.255986 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-xks-ca" pod "ab07e89d-1bba-4db9-97e2-e84dd54ca160" (UID: "ab07e89d-1bba-4db9-97e2-e84dd54ca160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:07:21.256120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.256101 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab07e89d-1bba-4db9-97e2-e84dd54ca160-kube-api-access-9drgt" pod "ab07e89d-1bba-4db9-97e2-e84dd54ca160" (UID: "ab07e89d-1bba-4db9-97e2-e84dd54ca160"). InnerVolumeSpecName "kube-api-access-9drgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:21.356567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.356531 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab07e89d-1bba-4db9-97e2-e84dd54ca160-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:21.356567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:21.356571 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9drgt\" (UniqueName: \"kubernetes.io/projected/ab07e89d-1bba-4db9-97e2-e84dd54ca160-kube-api-access-9drgt\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:22.010088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:22.010049 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:22.034746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:22.034686 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" Oct 30 01:07:22.034908 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:22.034678 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr" event={"ID":"ab07e89d-1bba-4db9-97e2-e84dd54ca160","Type":"ContainerDied","Data":"b3e896ce582099d7758c4199f4e47539a3f738086d875ee790f8b61bf663d74f"} Oct 30 01:07:22.034908 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:22.034849 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3e896ce582099d7758c4199f4e47539a3f738086d875ee790f8b61bf663d74f" Oct 30 01:07:22.692409 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:22.692372 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:22.804332 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:22.804303 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:23.978966 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:23.978929 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:25.967785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:25.967754 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:26.279414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.279323 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv"] Oct 30 01:07:26.361758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.357194 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:26.915341 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.915307 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w"] Oct 30 01:07:26.916028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.916012 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ab07e89d-1bba-4db9-97e2-e84dd54ca160" containerName="revoke-keys" Oct 30 01:07:26.916079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.916029 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab07e89d-1bba-4db9-97e2-e84dd54ca160" containerName="revoke-keys" Oct 30 01:07:26.916130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.916121 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab07e89d-1bba-4db9-97e2-e84dd54ca160" containerName="revoke-keys" Oct 30 01:07:26.921423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.921404 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:26.940031 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.940002 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w"] Oct 30 01:07:26.945027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.944999 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:07:26.945211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:26.945062 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:07:27.000736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.000688 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4cce7fe-kdc9w\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.001201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.000829 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggjb\" (UniqueName: \"kubernetes.io/projected/30359cf3-e8e1-4df3-9f28-870c3fed5203-kube-api-access-sggjb\") pod \"maas-api-revoke-sub-b4cce7fe-kdc9w\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.001201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.000993 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4cce7fe-kdc9w\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.102533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.102493 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4cce7fe-kdc9w\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.102744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.102578 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggjb\" (UniqueName: \"kubernetes.io/projected/30359cf3-e8e1-4df3-9f28-870c3fed5203-kube-api-access-sggjb\") pod \"maas-api-revoke-sub-b4cce7fe-kdc9w\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.102744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.102687 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4cce7fe-kdc9w\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.103355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.103325 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4cce7fe-kdc9w\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.105430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.105401 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4cce7fe-kdc9w\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.126846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.126823 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggjb\" (UniqueName: \"kubernetes.io/projected/30359cf3-e8e1-4df3-9f28-870c3fed5203-kube-api-access-sggjb\") pod \"maas-api-revoke-sub-b4cce7fe-kdc9w\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.235920 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.235831 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:27.582000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:27.581973 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w"] Oct 30 01:07:27.582664 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:07:27.582638 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30359cf3_e8e1_4df3_9f28_870c3fed5203.slice/crio-5a152ee96dc81d06126eb91b8af6d070334cde6a8b4dea2e2d4736450663d457 WatchSource:0}: Error finding container 5a152ee96dc81d06126eb91b8af6d070334cde6a8b4dea2e2d4736450663d457: Status 404 returned error can't find the container with id 5a152ee96dc81d06126eb91b8af6d070334cde6a8b4dea2e2d4736450663d457 Oct 30 01:07:28.023742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:28.023661 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:28.153489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:28.153392 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:28.161889 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:28.161765 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" event={"ID":"30359cf3-e8e1-4df3-9f28-870c3fed5203","Type":"ContainerStarted","Data":"832b5b4c4d266606c925136d4671b5880ef2a88857c0b19ed9a7fde0b797ccbb"} Oct 30 01:07:28.162181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:28.162160 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" event={"ID":"30359cf3-e8e1-4df3-9f28-870c3fed5203","Type":"ContainerStarted","Data":"5a152ee96dc81d06126eb91b8af6d070334cde6a8b4dea2e2d4736450663d457"} Oct 30 01:07:28.199445 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:28.199395 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" podStartSLOduration=2.199380351 podStartE2EDuration="2.199380351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:28.193358945 +0000 UTC m=+1986.105486085" watchObservedRunningTime="2026-10-30 01:07:28.199380351 +0000 UTC m=+1986.111507491" Oct 30 01:07:28.382471 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:28.382428 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:29.168614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:29.168577 4517 generic.go:356] "Generic (PLEG): container finished" podID="30359cf3-e8e1-4df3-9f28-870c3fed5203" containerID="832b5b4c4d266606c925136d4671b5880ef2a88857c0b19ed9a7fde0b797ccbb" exitCode=0 Oct 30 01:07:29.169107 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:29.168663 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" event={"ID":"30359cf3-e8e1-4df3-9f28-870c3fed5203","Type":"ContainerDied","Data":"832b5b4c4d266606c925136d4671b5880ef2a88857c0b19ed9a7fde0b797ccbb"} Oct 30 01:07:30.125269 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.125240 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:30.429693 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.429667 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:30.568330 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.568300 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-xks-ca\") pod \"30359cf3-e8e1-4df3-9f28-870c3fed5203\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " Oct 30 01:07:30.568544 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.568461 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30359cf3-e8e1-4df3-9f28-870c3fed5203-kube-api-access-sggjb\" (UniqueName: \"kubernetes.io/projected/30359cf3-e8e1-4df3-9f28-870c3fed5203-kube-api-access-sggjb\") pod \"30359cf3-e8e1-4df3-9f28-870c3fed5203\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " Oct 30 01:07:30.568626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.568592 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-openshift-ca\") pod \"30359cf3-e8e1-4df3-9f28-870c3fed5203\" (UID: \"30359cf3-e8e1-4df3-9f28-870c3fed5203\") " Oct 30 01:07:30.569099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.569068 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-openshift-ca" pod "30359cf3-e8e1-4df3-9f28-870c3fed5203" (UID: "30359cf3-e8e1-4df3-9f28-870c3fed5203"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:30.569378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.569361 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:30.571058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.570995 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-xks-ca" pod "30359cf3-e8e1-4df3-9f28-870c3fed5203" (UID: "30359cf3-e8e1-4df3-9f28-870c3fed5203"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:07:30.571199 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.571176 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30359cf3-e8e1-4df3-9f28-870c3fed5203-kube-api-access-sggjb" pod "30359cf3-e8e1-4df3-9f28-870c3fed5203" (UID: "30359cf3-e8e1-4df3-9f28-870c3fed5203"). InnerVolumeSpecName "kube-api-access-sggjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:30.671429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.671338 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sggjb\" (UniqueName: \"kubernetes.io/projected/30359cf3-e8e1-4df3-9f28-870c3fed5203-kube-api-access-sggjb\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:30.671429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:30.671379 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30359cf3-e8e1-4df3-9f28-870c3fed5203-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:30.761670 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:07:30.761631 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:07:31.203693 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:31.203659 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" Oct 30 01:07:31.203879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:31.203695 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w" event={"ID":"30359cf3-e8e1-4df3-9f28-870c3fed5203","Type":"ContainerDied","Data":"5a152ee96dc81d06126eb91b8af6d070334cde6a8b4dea2e2d4736450663d457"} Oct 30 01:07:31.203879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:31.203760 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a152ee96dc81d06126eb91b8af6d070334cde6a8b4dea2e2d4736450663d457" Oct 30 01:07:31.732158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:31.732122 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:33.397072 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:33.397029 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:34.957344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:34.957312 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:36.517257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:36.517223 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:38.358585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:38.358547 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:38.949278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:38.949237 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:39.066614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:39.066573 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:39.351047 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:39.351011 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg"] Oct 30 01:07:39.473782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:39.472168 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:40.186456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:40.186417 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:41.302402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.302350 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" podUID="a7049336-fa54-49d4-9d60-4bd14f9b5ae2" containerName="main" containerID="cri-o://7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574" gracePeriod=45 Oct 30 01:07:41.615309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.615280 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:07:41.666792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.666751 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-kube-api-access-7c8lx\" (UniqueName: \"kubernetes.io/projected/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-kube-api-access-7c8lx\") pod \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " Oct 30 01:07:41.666998 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.666813 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-model-cache\") pod \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " Oct 30 01:07:41.666998 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.666951 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-dshm\") pod \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " Oct 30 01:07:41.667140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.667064 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tls-certs\") pod \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " Oct 30 01:07:41.668120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.667226 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tmp-dir\") pod \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " Oct 30 01:07:41.668120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.667364 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-home\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-home\") pod \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\" (UID: \"a7049336-fa54-49d4-9d60-4bd14f9b5ae2\") " Oct 30 01:07:41.668120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.667466 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-model-cache" pod "a7049336-fa54-49d4-9d60-4bd14f9b5ae2" (UID: "a7049336-fa54-49d4-9d60-4bd14f9b5ae2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:41.668120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.667746 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tmp-dir" pod "a7049336-fa54-49d4-9d60-4bd14f9b5ae2" (UID: "a7049336-fa54-49d4-9d60-4bd14f9b5ae2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:41.668120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.667973 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-home" pod "a7049336-fa54-49d4-9d60-4bd14f9b5ae2" (UID: "a7049336-fa54-49d4-9d60-4bd14f9b5ae2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:41.668120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.668054 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:41.668120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.668073 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:41.671000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.670973 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-dshm" pod "a7049336-fa54-49d4-9d60-4bd14f9b5ae2" (UID: "a7049336-fa54-49d4-9d60-4bd14f9b5ae2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:41.679614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.679554 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tls-certs" pod "a7049336-fa54-49d4-9d60-4bd14f9b5ae2" (UID: "a7049336-fa54-49d4-9d60-4bd14f9b5ae2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:07:41.686325 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.686298 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-kube-api-access-7c8lx" pod "a7049336-fa54-49d4-9d60-4bd14f9b5ae2" (UID: "a7049336-fa54-49d4-9d60-4bd14f9b5ae2"). InnerVolumeSpecName "kube-api-access-7c8lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:41.770053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.770008 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:41.770053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.770040 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:41.770053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.770050 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8lx\" (UniqueName: \"kubernetes.io/projected/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-kube-api-access-7c8lx\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:41.770053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.770059 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7049336-fa54-49d4-9d60-4bd14f9b5ae2-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:41.961032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:41.960945 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:42.306734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:42.305739 4517 generic.go:356] "Generic (PLEG): container finished" podID="a7049336-fa54-49d4-9d60-4bd14f9b5ae2" containerID="7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574" exitCode=0 Oct 30 01:07:42.306734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:42.305891 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" event={"ID":"a7049336-fa54-49d4-9d60-4bd14f9b5ae2","Type":"ContainerDied","Data":"7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574"} Oct 30 01:07:42.306734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:42.305940 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" event={"ID":"a7049336-fa54-49d4-9d60-4bd14f9b5ae2","Type":"ContainerDied","Data":"3be97f3a9e0b0235c20b85d1dc2e4198acd8a166e2df3231d12cc53f6f3a1404"} Oct 30 01:07:42.306734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:42.305962 4517 scope.go:122] "RemoveContainer" containerID="7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574" Oct 30 01:07:42.306734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:42.306512 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv" Oct 30 01:07:42.317299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:42.317276 4517 scope.go:122] "RemoveContainer" containerID="7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574" Oct 30 01:07:42.317603 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:07:42.317580 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574\": container with ID starting with 7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574 not found: ID does not exist" containerID="7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574" Oct 30 01:07:42.317651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:42.317612 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574"} err="failed to get container status \"7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574\": rpc error: code = NotFound desc = could not find container \"7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574\": container with ID starting with 7856c22ee0bb03523f9a0ad3608a7f66336590917b48cb097ec63cac71694574 not found: ID does not exist" Oct 30 01:07:42.353850 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:42.353818 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv"] Oct 30 01:07:42.366622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:42.366584 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-820c04/test-model-820c04-kserve-6cc6cc7569-ttdkv"] Oct 30 01:07:43.705412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:43.705366 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:44.237258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:44.237226 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7049336-fa54-49d4-9d60-4bd14f9b5ae2" path="/var/lib/kubelet/pods/a7049336-fa54-49d4-9d60-4bd14f9b5ae2/volumes" Oct 30 01:07:45.366140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:45.366100 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:47.313868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:47.313837 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:47.879922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:47.879890 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:48.001882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:48.001848 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:49.124260 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:49.124226 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:50.189283 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:50.189230 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:50.332829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:50.327615 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:50.829313 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:50.829273 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:52.246720 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:52.246661 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:52.847474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:52.847404 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:53.002612 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:53.002549 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:53.026618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:53.026585 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg"] Oct 30 01:07:53.036282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:53.036253 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a07920-8hrcg"] Oct 30 01:07:53.119492 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:53.119397 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:53.509297 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:53.509215 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:54.239305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.239270 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="104d83cc-f44d-40c3-aaf6-f7e91c2f498c" path="/var/lib/kubelet/pods/104d83cc-f44d-40c3-aaf6-f7e91c2f498c/volumes" Oct 30 01:07:54.375336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.375290 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" containerID="cri-o://aef5cf916476508e3850b7aa59480c0cab34dc52f14593b243fa9ade49f460a6" gracePeriod=45 Oct 30 01:07:54.496015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.495983 4517 generic.go:356] "Generic (PLEG): container finished" podID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerID="aef5cf916476508e3850b7aa59480c0cab34dc52f14593b243fa9ade49f460a6" exitCode=0 Oct 30 01:07:54.496148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.496058 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" event={"ID":"3dae4e81-bfe5-46bf-977c-a7fa6412c48a","Type":"ContainerDied","Data":"aef5cf916476508e3850b7aa59480c0cab34dc52f14593b243fa9ade49f460a6"} Oct 30 01:07:54.672722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.672679 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:07:54.736753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.736677 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:54.822620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.822563 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-kube-api-access-bt2tc\" (UniqueName: \"kubernetes.io/projected/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-kube-api-access-bt2tc\") pod \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " Oct 30 01:07:54.822840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.822670 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-dshm\") pod \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " Oct 30 01:07:54.822840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.822746 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-home\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-home\") pod \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " Oct 30 01:07:54.822840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.822828 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-model-cache\") pod \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " Oct 30 01:07:54.822965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.822875 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tmp-dir\") pod \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " Oct 30 01:07:54.822965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.822920 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tls-certs\" (UniqueName: \"kubernetes.io/secret/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tls-certs\") pod \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\" (UID: \"3dae4e81-bfe5-46bf-977c-a7fa6412c48a\") " Oct 30 01:07:54.824470 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.824431 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-home" pod "3dae4e81-bfe5-46bf-977c-a7fa6412c48a" (UID: "3dae4e81-bfe5-46bf-977c-a7fa6412c48a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:54.825134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.824937 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-model-cache" pod "3dae4e81-bfe5-46bf-977c-a7fa6412c48a" (UID: "3dae4e81-bfe5-46bf-977c-a7fa6412c48a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:54.825134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.825094 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tmp-dir" pod "3dae4e81-bfe5-46bf-977c-a7fa6412c48a" (UID: "3dae4e81-bfe5-46bf-977c-a7fa6412c48a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:54.828767 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.828018 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-kube-api-access-bt2tc" pod "3dae4e81-bfe5-46bf-977c-a7fa6412c48a" (UID: "3dae4e81-bfe5-46bf-977c-a7fa6412c48a"). InnerVolumeSpecName "kube-api-access-bt2tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:54.828896 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.828810 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tls-certs" pod "3dae4e81-bfe5-46bf-977c-a7fa6412c48a" (UID: "3dae4e81-bfe5-46bf-977c-a7fa6412c48a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:07:54.829921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.829894 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-dshm" pod "3dae4e81-bfe5-46bf-977c-a7fa6412c48a" (UID: "3dae4e81-bfe5-46bf-977c-a7fa6412c48a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:54.925611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.925575 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt2tc\" (UniqueName: \"kubernetes.io/projected/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-kube-api-access-bt2tc\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:54.925611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.925605 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:54.925611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.925618 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:54.925921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.925626 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:54.925921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.925639 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:54.925921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:54.925646 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dae4e81-bfe5-46bf-977c-a7fa6412c48a-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:07:55.507909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:55.507877 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" Oct 30 01:07:55.508127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:55.507887 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg" event={"ID":"3dae4e81-bfe5-46bf-977c-a7fa6412c48a","Type":"ContainerDied","Data":"ca35d1fc4cd0f1bf6b8d956e7d27c6e07ca67260f268281303ebc636fab98b70"} Oct 30 01:07:55.508127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:55.508002 4517 scope.go:122] "RemoveContainer" containerID="aef5cf916476508e3850b7aa59480c0cab34dc52f14593b243fa9ade49f460a6" Oct 30 01:07:55.555682 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:55.555652 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg"] Oct 30 01:07:55.566389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:55.566356 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-dd257f/test-model-dd257f-kserve-748c8c878c-xfffg"] Oct 30 01:07:56.237053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:56.237007 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" path="/var/lib/kubelet/pods/3dae4e81-bfe5-46bf-977c-a7fa6412c48a/volumes" Oct 30 01:07:56.623947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:56.623916 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:58.408786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:58.408703 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:59.832600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:59.832565 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:59.947221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:07:59.947188 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:00.062168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:00.062130 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:00.145629 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:00.145488 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44"] Oct 30 01:08:00.147861 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:00.146200 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" containerID="cri-o://dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9" gracePeriod=30 Oct 30 01:08:00.162886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:00.160450 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9"] Oct 30 01:08:00.162886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:00.162140 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerName="istio-proxy" containerID="cri-o://44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0" gracePeriod=30 Oct 30 01:08:00.195835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:00.195760 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:00.761537 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:08:00.761488 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:08:01.837137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:01.837099 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:03.490518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:03.490480 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:04.896473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:04.896448 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:05.479739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.462475 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:05.657405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.657360 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:05.696196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.696132 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr"] Oct 30 01:08:05.704548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.704368 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8b26-5ztdr"] Oct 30 01:08:05.869283 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.869213 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:08:05.871404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.871382 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:08:05.894869 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.894812 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:05.952063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952033 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-kube-api-access-hnkl7\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-kube-api-access-hnkl7\") pod \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " Oct 30 01:08:05.952063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952069 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-credential-socket\") pod \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " Oct 30 01:08:05.952623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952115 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-data\") pod \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " Oct 30 01:08:05.952623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952363 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istiod-ca-cert\") pod \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " Oct 30 01:08:05.952623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952415 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-podinfo\") pod \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " Oct 30 01:08:05.952623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952506 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-certs\") pod \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " Oct 30 01:08:05.952623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952559 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-certs\") pod \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " Oct 30 01:08:05.952623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952598 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-envoy\") pod \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " Oct 30 01:08:05.952985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952787 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istiod-ca-cert\") pod \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " Oct 30 01:08:05.952985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952820 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-envoy\") pod \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " Oct 30 01:08:05.952985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952487 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-data" pod "bcd451a3-1147-45c1-acc3-628f6fa6ac4e" (UID: "bcd451a3-1147-45c1-acc3-628f6fa6ac4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.952985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952802 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istiod-ca-cert" pod "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" (UID: "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:05.952985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952853 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-podinfo\") pod \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " Oct 30 01:08:05.952985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952880 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-data\") pod \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " Oct 30 01:08:05.952985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952918 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-credential-socket\") pod \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " Oct 30 01:08:05.952985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952949 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-socket\") pod \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " Oct 30 01:08:05.952985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.952979 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-token\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-token\") pod \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\" (UID: \"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f\") " Oct 30 01:08:05.953866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.953084 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-certs" pod "bcd451a3-1147-45c1-acc3-628f6fa6ac4e" (UID: "bcd451a3-1147-45c1-acc3-628f6fa6ac4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.953866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.953094 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-kube-api-access-8wv2w\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-kube-api-access-8wv2w\") pod \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " Oct 30 01:08:05.953866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.953168 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-socket\") pod \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " Oct 30 01:08:05.953866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.953432 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istiod-ca-cert" pod "bcd451a3-1147-45c1-acc3-628f6fa6ac4e" (UID: "bcd451a3-1147-45c1-acc3-628f6fa6ac4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:05.954094 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.953212 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-token\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-token\") pod \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\" (UID: \"bcd451a3-1147-45c1-acc3-628f6fa6ac4e\") " Oct 30 01:08:05.954646 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.954378 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-credential-socket" pod "bcd451a3-1147-45c1-acc3-628f6fa6ac4e" (UID: "bcd451a3-1147-45c1-acc3-628f6fa6ac4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.954819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.954795 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-socket" pod "bcd451a3-1147-45c1-acc3-628f6fa6ac4e" (UID: "bcd451a3-1147-45c1-acc3-628f6fa6ac4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.957324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.957218 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.957324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.957244 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.957324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.957259 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.957324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.957272 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.957324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.957285 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.957324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.957297 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.958533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.957851 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-kube-api-access-hnkl7" pod "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" (UID: "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f"). InnerVolumeSpecName "kube-api-access-hnkl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:05.958533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.958245 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-kube-api-access-8wv2w" pod "bcd451a3-1147-45c1-acc3-628f6fa6ac4e" (UID: "bcd451a3-1147-45c1-acc3-628f6fa6ac4e"). InnerVolumeSpecName "kube-api-access-8wv2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:05.960133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.960002 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-socket" pod "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" (UID: "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.960133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.960021 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-podinfo" pod "bcd451a3-1147-45c1-acc3-628f6fa6ac4e" (UID: "bcd451a3-1147-45c1-acc3-628f6fa6ac4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:05.960133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.960062 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-credential-socket" pod "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" (UID: "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.960133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.960108 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-data" pod "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" (UID: "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.960555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.960531 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-certs" pod "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" (UID: "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.961960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.961264 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-envoy" pod "bcd451a3-1147-45c1-acc3-628f6fa6ac4e" (UID: "bcd451a3-1147-45c1-acc3-628f6fa6ac4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.961960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.961357 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-podinfo" pod "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" (UID: "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:05.963109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.963071 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-token" pod "bcd451a3-1147-45c1-acc3-628f6fa6ac4e" (UID: "bcd451a3-1147-45c1-acc3-628f6fa6ac4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:05.967576 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.967551 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-envoy" pod "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" (UID: "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.969680 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:05.969656 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-token" pod "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" (UID: "00d1e1f9-2ec5-4325-a8e1-feba17d55a3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:06.008389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.008337 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:06.058945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.058902 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.058945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.058939 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.058955 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.058974 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.058990 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.059003 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.059017 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wv2w\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-kube-api-access-8wv2w\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.059034 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.059048 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkl7\" (UniqueName: \"kubernetes.io/projected/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-kube-api-access-hnkl7\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.059065 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.059078 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.059168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.059092 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd451a3-1147-45c1-acc3-628f6fa6ac4e-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.132838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.132644 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:06.237070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.237036 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab07e89d-1bba-4db9-97e2-e84dd54ca160" path="/var/lib/kubelet/pods/ab07e89d-1bba-4db9-97e2-e84dd54ca160/volumes" Oct 30 01:08:06.251890 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.251233 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:06.615480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.615445 4517 generic.go:356] "Generic (PLEG): container finished" podID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerID="44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0" exitCode=0 Oct 30 01:08:06.615683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.615491 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" Oct 30 01:08:06.615683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.615533 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" event={"ID":"bcd451a3-1147-45c1-acc3-628f6fa6ac4e","Type":"ContainerDied","Data":"44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0"} Oct 30 01:08:06.615683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.615598 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9" event={"ID":"bcd451a3-1147-45c1-acc3-628f6fa6ac4e","Type":"ContainerDied","Data":"2331dd8c8a7cd326f53f068a6c21cd3b26d96dc9e8bfd754e03befdf84ff1e3d"} Oct 30 01:08:06.615683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.615630 4517 scope.go:122] "RemoveContainer" containerID="44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0" Oct 30 01:08:06.617330 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.617309 4517 generic.go:356] "Generic (PLEG): container finished" podID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerID="dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9" exitCode=0 Oct 30 01:08:06.617452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.617360 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" Oct 30 01:08:06.617452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.617370 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" event={"ID":"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f","Type":"ContainerDied","Data":"dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9"} Oct 30 01:08:06.617452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.617412 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44" event={"ID":"00d1e1f9-2ec5-4325-a8e1-feba17d55a3f","Type":"ContainerDied","Data":"2057c7a09f26489c1a0c2b6b31fe9fe0f879187f3dfd2fdcfcb43c2d9a3d14c1"} Oct 30 01:08:06.626750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.626695 4517 scope.go:122] "RemoveContainer" containerID="44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0" Oct 30 01:08:06.627070 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:08:06.627052 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0\": container with ID starting with 44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0 not found: ID does not exist" containerID="44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0" Oct 30 01:08:06.627148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.627078 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0"} err="failed to get container status \"44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0\": rpc error: code = NotFound desc = could not find container \"44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0\": container with ID starting with 44ef0fd7d917988db0abe8b9336fba20e26d0f879b0fc87d6a58015c9ff2b7c0 not found: ID does not exist" Oct 30 01:08:06.627148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.627095 4517 scope.go:122] "RemoveContainer" containerID="dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9" Oct 30 01:08:06.640571 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.640208 4517 scope.go:122] "RemoveContainer" containerID="dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9" Oct 30 01:08:06.640922 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:08:06.640891 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9\": container with ID starting with dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9 not found: ID does not exist" containerID="dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9" Oct 30 01:08:06.641018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.640931 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9"} err="failed to get container status \"dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9\": rpc error: code = NotFound desc = could not find container \"dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9\": container with ID starting with dd5fcad56a559ccad795cce48a7beb176d5598614621f2dce90c500015380ed9 not found: ID does not exist" Oct 30 01:08:06.666071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.666026 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9"] Oct 30 01:08:06.676420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.676387 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-dvkv9"] Oct 30 01:08:06.699050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.699011 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44"] Oct 30 01:08:06.704690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:06.704661 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-mzh44"] Oct 30 01:08:07.884450 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:07.884414 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:08.239027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:08.238940 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" path="/var/lib/kubelet/pods/00d1e1f9-2ec5-4325-a8e1-feba17d55a3f/volumes" Oct 30 01:08:08.239538 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:08.239524 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" path="/var/lib/kubelet/pods/bcd451a3-1147-45c1-acc3-628f6fa6ac4e/volumes" Oct 30 01:08:09.637822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:09.637779 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:11.292600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.292565 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.756756 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5"] Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757531 4517 cpu_manager.go:414] "containerMap: removing container" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757553 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757578 4517 cpu_manager.go:414] "containerMap: removing container" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757584 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757604 4517 cpu_manager.go:414] "containerMap: removing container" podUID="30359cf3-e8e1-4df3-9f28-870c3fed5203" containerName="revoke-keys" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757608 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30359cf3-e8e1-4df3-9f28-870c3fed5203" containerName="revoke-keys" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757629 4517 cpu_manager.go:414] "containerMap: removing container" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerName="istio-proxy" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757635 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerName="istio-proxy" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757647 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a7049336-fa54-49d4-9d60-4bd14f9b5ae2" containerName="main" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757651 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7049336-fa54-49d4-9d60-4bd14f9b5ae2" containerName="main" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757803 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd451a3-1147-45c1-acc3-628f6fa6ac4e" containerName="istio-proxy" Oct 30 01:08:11.757809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757820 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7049336-fa54-49d4-9d60-4bd14f9b5ae2" containerName="main" Oct 30 01:08:11.758582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757832 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dae4e81-bfe5-46bf-977c-a7fa6412c48a" containerName="main" Oct 30 01:08:11.758582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757844 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d1e1f9-2ec5-4325-a8e1-feba17d55a3f" containerName="istio-proxy" Oct 30 01:08:11.758582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.757858 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="30359cf3-e8e1-4df3-9f28-870c3fed5203" containerName="revoke-keys" Oct 30 01:08:11.763332 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.763307 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.788278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.787634 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-95492e\"/\"kube-root-ca.crt\"" Oct 30 01:08:11.788278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.787648 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-95492e\"/\"default-dockercfg-9gpmj\"" Oct 30 01:08:11.788551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.788345 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-95492e\"/\"auth-test-model-95492e-kserve-self-signed-certs\"" Oct 30 01:08:11.788551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.788412 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-95492e\"/\"openshift-service-ca.crt\"" Oct 30 01:08:11.794555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.794510 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5"] Oct 30 01:08:11.857375 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.857337 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:11.858256 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.858223 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-model-cache\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.858672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.858641 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-home\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.858875 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.858699 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntmp\" (UniqueName: \"kubernetes.io/projected/da946f82-5476-4580-b606-53151cf7fa9d-kube-api-access-nntmp\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.858946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.858891 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-dshm\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.859024 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.859004 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da946f82-5476-4580-b606-53151cf7fa9d-tls-certs\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.859110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.859095 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-tmp-dir\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.961275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.961226 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-dshm\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.961494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.961342 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da946f82-5476-4580-b606-53151cf7fa9d-tls-certs\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.961494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.961434 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-tmp-dir\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.961627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.961555 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-model-cache\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.961955 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.961930 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-home\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.962045 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.961965 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-tmp-dir\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.962106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.962029 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-model-cache\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.962238 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.962214 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-home\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.962331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.962308 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntmp\" (UniqueName: \"kubernetes.io/projected/da946f82-5476-4580-b606-53151cf7fa9d-kube-api-access-nntmp\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.964040 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.964011 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-dshm\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.964765 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.964701 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da946f82-5476-4580-b606-53151cf7fa9d-tls-certs\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:11.986819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:11.986777 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntmp\" (UniqueName: \"kubernetes.io/projected/da946f82-5476-4580-b606-53151cf7fa9d-kube-api-access-nntmp\") pod \"auth-test-model-95492e-kserve-84fd8d947c-tqnq5\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:12.091734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:12.084037 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:12.529968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:12.529919 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5"] Oct 30 01:08:12.532054 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:08:12.532027 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda946f82_5476_4580_b606_53151cf7fa9d.slice/crio-301af0b1c97208526653738f010b4b76d7a974d8231632227a242ee638cc7c0f WatchSource:0}: Error finding container 301af0b1c97208526653738f010b4b76d7a974d8231632227a242ee638cc7c0f: Status 404 returned error can't find the container with id 301af0b1c97208526653738f010b4b76d7a974d8231632227a242ee638cc7c0f Oct 30 01:08:12.667630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:12.667592 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" event={"ID":"da946f82-5476-4580-b606-53151cf7fa9d","Type":"ContainerStarted","Data":"35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a"} Oct 30 01:08:12.667915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:12.667639 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" event={"ID":"da946f82-5476-4580-b606-53151cf7fa9d","Type":"ContainerStarted","Data":"301af0b1c97208526653738f010b4b76d7a974d8231632227a242ee638cc7c0f"} Oct 30 01:08:12.717104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:12.717052 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" podStartSLOduration=1.717036995 podStartE2EDuration="1.717036995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:12.71394565 +0000 UTC m=+2030.626072792" watchObservedRunningTime="2026-10-30 01:08:12.717036995 +0000 UTC m=+2030.629164136" Oct 30 01:08:13.085373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:13.085338 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:13.320751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:13.320686 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:13.502249 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:13.502166 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:13.623985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:13.623938 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:13.951244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:13.951210 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:14.635958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:14.635915 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:15.121090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:15.121039 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:15.263735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:15.246498 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:16.583047 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:16.583005 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:16.819509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:16.819467 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:18.591916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:18.591875 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:18.745129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:18.745082 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:18.872786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:18.872590 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:20.385518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:20.385472 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:20.508135 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:20.508100 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:21.443590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:21.443550 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:22.084929 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:22.084893 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:22.098489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:22.098463 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:22.515145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:22.515047 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:22.912208 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:22.912178 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:08:23.259818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:23.259728 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:24.826235 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:24.826202 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:26.519340 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:26.519292 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:28.200546 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:28.200507 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:29.822449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:29.822415 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:30.761583 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:08:30.761538 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:08:31.718204 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:31.718166 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:33.137162 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:33.137088 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:33.986776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:33.986739 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:34.122612 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:34.122575 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:34.244167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:34.244058 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:34.612203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:34.612165 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:36.070419 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:36.070373 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:36.283783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:36.283734 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj"] Oct 30 01:08:36.340508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:36.340413 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:37.782267 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:37.782229 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:39.381398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:39.381360 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx"] Oct 30 01:08:39.381873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:39.381652 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" containerID="cri-o://5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc" gracePeriod=30 Oct 30 01:08:39.393229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:39.393189 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:39.513360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:39.513317 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:39.638442 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:39.638341 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:41.122616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:41.122579 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:41.250510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:41.250476 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:41.467018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:41.466907 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:41.612376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:41.612338 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:42.565261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:42.565218 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:43.484972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:43.484914 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:45.158858 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.158808 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 01:08:45.164837 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.164803 4517 generic.go:356] "Generic (PLEG): container finished" podID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerID="5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc" exitCode=0 Oct 30 01:08:45.165005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.164868 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" event={"ID":"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8","Type":"ContainerDied","Data":"5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc"} Oct 30 01:08:45.165005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.164916 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" event={"ID":"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8","Type":"ContainerDied","Data":"330bffd75fdd4e4646bfa32aec6d9276664adeee910be3452d02717b3e60f032"} Oct 30 01:08:45.165005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.164944 4517 scope.go:122] "RemoveContainer" containerID="5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc" Oct 30 01:08:45.165190 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.165104 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx" Oct 30 01:08:45.182950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.182925 4517 scope.go:122] "RemoveContainer" containerID="5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc" Oct 30 01:08:45.183432 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:08:45.183399 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc\": container with ID starting with 5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc not found: ID does not exist" containerID="5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc" Oct 30 01:08:45.183572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.183445 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc"} err="failed to get container status \"5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc\": rpc error: code = NotFound desc = could not find container \"5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc\": container with ID starting with 5f63f9fa62b6def7b1f809af663056c936f561c780d3398ffaabaea6fb9b83bc not found: ID does not exist" Oct 30 01:08:45.218253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.218193 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-data\") pod \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " Oct 30 01:08:45.219028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.218849 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-data" pod "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" (UID: "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:45.219463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.219442 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-token\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-token\") pod \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " Oct 30 01:08:45.219651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.219636 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-envoy\") pod \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " Oct 30 01:08:45.219853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.219834 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-socket\") pod \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " Oct 30 01:08:45.220051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.220015 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istiod-ca-cert\") pod \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " Oct 30 01:08:45.220533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.220514 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-podinfo\") pod \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " Oct 30 01:08:45.220852 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.220772 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-credential-socket\") pod \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " Oct 30 01:08:45.221115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.221096 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-kube-api-access-5grr6\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-kube-api-access-5grr6\") pod \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " Oct 30 01:08:45.221323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.221306 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-certs\") pod \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\" (UID: \"9f8a756b-d0ec-4fec-952c-4ae5e28b54f8\") " Oct 30 01:08:45.222145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.222121 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:45.224292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.223479 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istiod-ca-cert" pod "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" (UID: "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:45.226058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.223751 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-credential-socket" pod "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" (UID: "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:45.226193 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.223784 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-envoy" pod "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" (UID: "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:45.226281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.224039 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-certs" pod "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" (UID: "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:45.226369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.224222 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-socket" pod "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" (UID: "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:45.228788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.225602 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-token" pod "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" (UID: "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:45.229516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.229490 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-kube-api-access-5grr6" pod "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" (UID: "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8"). InnerVolumeSpecName "kube-api-access-5grr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:45.229868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.229839 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-podinfo" pod "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" (UID: "9f8a756b-d0ec-4fec-952c-4ae5e28b54f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:45.357765 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.346930 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:45.376024 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.375939 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:45.376262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.376243 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:45.376379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.376364 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:45.376479 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.376467 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:45.376568 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.376556 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:45.376651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.376640 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5grr6\" (UniqueName: \"kubernetes.io/projected/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-kube-api-access-5grr6\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:45.376763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.376748 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:45.535915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.535877 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx"] Oct 30 01:08:45.539932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.539896 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zw4lx"] Oct 30 01:08:45.919456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:45.919420 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:46.236901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:46.236816 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" path="/var/lib/kubelet/pods/9f8a756b-d0ec-4fec-952c-4ae5e28b54f8/volumes" Oct 30 01:08:47.812352 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:47.812308 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:49.394897 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:49.394858 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:51.072086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.072046 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:51.336118 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.336014 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" containerID="cri-o://d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6" gracePeriod=45 Oct 30 01:08:51.660256 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.660224 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:08:51.695823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.695784 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-home\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-home\") pod \"cd017f4b-0b2f-409b-a895-79c151564111\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " Oct 30 01:08:51.696089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.695918 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-dshm\") pod \"cd017f4b-0b2f-409b-a895-79c151564111\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " Oct 30 01:08:51.696089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.695998 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd017f4b-0b2f-409b-a895-79c151564111-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd017f4b-0b2f-409b-a895-79c151564111-tls-certs\") pod \"cd017f4b-0b2f-409b-a895-79c151564111\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " Oct 30 01:08:51.696089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.696034 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd017f4b-0b2f-409b-a895-79c151564111-kube-api-access-zft68\" (UniqueName: \"kubernetes.io/projected/cd017f4b-0b2f-409b-a895-79c151564111-kube-api-access-zft68\") pod \"cd017f4b-0b2f-409b-a895-79c151564111\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " Oct 30 01:08:51.696274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.696106 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-home" pod "cd017f4b-0b2f-409b-a895-79c151564111" (UID: "cd017f4b-0b2f-409b-a895-79c151564111"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:51.696274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.696144 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-tmp-dir\") pod \"cd017f4b-0b2f-409b-a895-79c151564111\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " Oct 30 01:08:51.696274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.696216 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-model-cache\") pod \"cd017f4b-0b2f-409b-a895-79c151564111\" (UID: \"cd017f4b-0b2f-409b-a895-79c151564111\") " Oct 30 01:08:51.696698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.696668 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-tmp-dir" pod "cd017f4b-0b2f-409b-a895-79c151564111" (UID: "cd017f4b-0b2f-409b-a895-79c151564111"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:51.696824 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.696781 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-model-cache" pod "cd017f4b-0b2f-409b-a895-79c151564111" (UID: "cd017f4b-0b2f-409b-a895-79c151564111"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:51.697348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.697309 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:51.697478 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.697339 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:51.697478 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.697366 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:51.698764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.698741 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-dshm" pod "cd017f4b-0b2f-409b-a895-79c151564111" (UID: "cd017f4b-0b2f-409b-a895-79c151564111"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:51.699104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.699084 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd017f4b-0b2f-409b-a895-79c151564111-tls-certs" pod "cd017f4b-0b2f-409b-a895-79c151564111" (UID: "cd017f4b-0b2f-409b-a895-79c151564111"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:51.699943 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.699478 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd017f4b-0b2f-409b-a895-79c151564111-kube-api-access-zft68" pod "cd017f4b-0b2f-409b-a895-79c151564111" (UID: "cd017f4b-0b2f-409b-a895-79c151564111"). InnerVolumeSpecName "kube-api-access-zft68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:51.814758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.805544 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd017f4b-0b2f-409b-a895-79c151564111-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:51.814758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.805577 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zft68\" (UniqueName: \"kubernetes.io/projected/cd017f4b-0b2f-409b-a895-79c151564111-kube-api-access-zft68\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:51.814758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.805592 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd017f4b-0b2f-409b-a895-79c151564111-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:08:51.842421 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:51.842355 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:52.239453 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.239415 4517 generic.go:356] "Generic (PLEG): container finished" podID="cd017f4b-0b2f-409b-a895-79c151564111" containerID="d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6" exitCode=0 Oct 30 01:08:52.239997 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.239567 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" Oct 30 01:08:52.248940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.248901 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" event={"ID":"cd017f4b-0b2f-409b-a895-79c151564111","Type":"ContainerDied","Data":"d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6"} Oct 30 01:08:52.249124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.248953 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj" event={"ID":"cd017f4b-0b2f-409b-a895-79c151564111","Type":"ContainerDied","Data":"0e274d24e8289474761a5dd63479a16ed76f30a6c8233f83401aea2487ef8dd6"} Oct 30 01:08:52.249124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.248977 4517 scope.go:122] "RemoveContainer" containerID="d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6" Oct 30 01:08:52.261209 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.261182 4517 scope.go:122] "RemoveContainer" containerID="d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6" Oct 30 01:08:52.261686 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:08:52.261655 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6\": container with ID starting with d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6 not found: ID does not exist" containerID="d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6" Oct 30 01:08:52.261838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.261697 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6"} err="failed to get container status \"d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6\": rpc error: code = NotFound desc = could not find container \"d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6\": container with ID starting with d68c0cb9dbfa8f173bcf372718389ed35882c26b81420ed90df24c5aa89d92e6 not found: ID does not exist" Oct 30 01:08:52.289755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.289701 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj"] Oct 30 01:08:52.299067 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.299043 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-780077/test-model-780077-kserve-755c9cc6c7-rhsgj"] Oct 30 01:08:52.681141 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.681109 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr"] Oct 30 01:08:52.716856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.716818 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb"] Oct 30 01:08:52.820264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.820223 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:52.945816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:52.943410 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:53.125783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:53.122891 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:53.563499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:53.563462 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn"] Oct 30 01:08:53.690235 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:53.690204 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:53.826207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:53.823390 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.273833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:54.265140 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd017f4b-0b2f-409b-a895-79c151564111" path="/var/lib/kubelet/pods/cd017f4b-0b2f-409b-a895-79c151564111/volumes" Oct 30 01:08:55.045606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:55.045575 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:56.774473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:56.774441 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:57.682433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:57.682389 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" podUID="c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" containerName="payload-processing" containerID="cri-o://c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f" gracePeriod=25 Oct 30 01:08:57.717720 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:57.717637 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" podUID="35c7e888-3215-475a-9c2e-4b4b9a9450df" containerName="payload-pre-processing" containerID="cri-o://28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d" gracePeriod=25 Oct 30 01:08:58.425265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:58.425228 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:58.564586 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:58.564516 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" podUID="97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" containerName="payload-pre-processing" containerID="cri-o://4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729" gracePeriod=25 Oct 30 01:08:58.776533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:58.776440 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" podUID="35c7e888-3215-475a-9c2e-4b4b9a9450df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:08:58.776533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:08:58.776451 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" podUID="c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:09:00.261903 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:00.261862 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:00.761059 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:09:00.761016 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:09:01.885740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:01.885664 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:02.020472 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:02.014058 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:02.174075 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:02.173967 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:02.545191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:02.545150 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:02.669735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:02.667331 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:03.815473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:03.815430 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:05.511573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:05.511533 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:07.169567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:07.169538 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:08.222562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:08.222514 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:08.248405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:08.248368 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" podUID="97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:09:08.776159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:08.776105 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" podUID="c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:09:08.776813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:08.776161 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" podUID="35c7e888-3215-475a-9c2e-4b4b9a9450df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:09:09.034101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:09.033957 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:10.814922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:10.814878 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:12.639984 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:12.639952 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:12.849239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:12.849200 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:12.968994 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:12.968888 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:14.230541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:14.230490 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:15.829691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:15.829657 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:16.708774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:16.708736 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:17.640351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:17.640300 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:18.051039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.050621 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:09:18.081876 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.081838 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:09:18.138824 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.138790 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-kube-api-access-gkqdj\" (UniqueName: \"kubernetes.io/projected/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-kube-api-access-gkqdj\") pod \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\" (UID: \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\") " Oct 30 01:09:18.139042 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.138963 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-plugins-config-volume\") pod \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\" (UID: \"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f\") " Oct 30 01:09:18.139665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.139633 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-plugins-config-volume" pod "c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" (UID: "c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:18.139887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.139867 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.141682 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.141649 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-kube-api-access-gkqdj" pod "c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" (UID: "c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f"). InnerVolumeSpecName "kube-api-access-gkqdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:18.240343 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.240256 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c7e888-3215-475a-9c2e-4b4b9a9450df-kube-api-access-vn998\" (UniqueName: \"kubernetes.io/projected/35c7e888-3215-475a-9c2e-4b4b9a9450df-kube-api-access-vn998\") pod \"35c7e888-3215-475a-9c2e-4b4b9a9450df\" (UID: \"35c7e888-3215-475a-9c2e-4b4b9a9450df\") " Oct 30 01:09:18.240343 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.240320 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c7e888-3215-475a-9c2e-4b4b9a9450df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35c7e888-3215-475a-9c2e-4b4b9a9450df-plugins-config-volume\") pod \"35c7e888-3215-475a-9c2e-4b4b9a9450df\" (UID: \"35c7e888-3215-475a-9c2e-4b4b9a9450df\") " Oct 30 01:09:18.240798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.240765 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c7e888-3215-475a-9c2e-4b4b9a9450df-plugins-config-volume" pod "35c7e888-3215-475a-9c2e-4b4b9a9450df" (UID: "35c7e888-3215-475a-9c2e-4b4b9a9450df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:18.240993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.240976 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35c7e888-3215-475a-9c2e-4b4b9a9450df-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.241040 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.240999 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqdj\" (UniqueName: \"kubernetes.io/projected/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f-kube-api-access-gkqdj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.242808 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.242782 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c7e888-3215-475a-9c2e-4b4b9a9450df-kube-api-access-vn998" pod "35c7e888-3215-475a-9c2e-4b4b9a9450df" (UID: "35c7e888-3215-475a-9c2e-4b4b9a9450df"). InnerVolumeSpecName "kube-api-access-vn998". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:18.248050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.248015 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" podUID="97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:09:18.342353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.342316 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn998\" (UniqueName: \"kubernetes.io/projected/35c7e888-3215-475a-9c2e-4b4b9a9450df-kube-api-access-vn998\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.551449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.551409 4517 generic.go:356] "Generic (PLEG): container finished" podID="35c7e888-3215-475a-9c2e-4b4b9a9450df" containerID="28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d" exitCode=0 Oct 30 01:09:18.551675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.551465 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" Oct 30 01:09:18.551675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.551494 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" event={"ID":"35c7e888-3215-475a-9c2e-4b4b9a9450df","Type":"ContainerDied","Data":"28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d"} Oct 30 01:09:18.551675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.551565 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb" event={"ID":"35c7e888-3215-475a-9c2e-4b4b9a9450df","Type":"ContainerDied","Data":"d3f64d741e8b1ff2bb86971e3139d501630c24baf4f65ec7b9de7b784fab5dde"} Oct 30 01:09:18.551675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.551595 4517 scope.go:122] "RemoveContainer" containerID="28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d" Oct 30 01:09:18.553339 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.553312 4517 generic.go:356] "Generic (PLEG): container finished" podID="c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" containerID="c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f" exitCode=0 Oct 30 01:09:18.553476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.553350 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" event={"ID":"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f","Type":"ContainerDied","Data":"c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f"} Oct 30 01:09:18.553476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.553352 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" Oct 30 01:09:18.553476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.553390 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr" event={"ID":"c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f","Type":"ContainerDied","Data":"d2161323c9d2158d246c199201b56b293e5024fb7a82217b0d494193678c51b5"} Oct 30 01:09:18.564254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.564228 4517 scope.go:122] "RemoveContainer" containerID="28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d" Oct 30 01:09:18.564789 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:09:18.564758 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d\": container with ID starting with 28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d not found: ID does not exist" containerID="28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d" Oct 30 01:09:18.564923 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.564804 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d"} err="failed to get container status \"28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d\": rpc error: code = NotFound desc = could not find container \"28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d\": container with ID starting with 28f8754adb6f27945820a3f127ff3a8586b1b22c610a1ad4a2820faa1c6ebd2d not found: ID does not exist" Oct 30 01:09:18.564923 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.564829 4517 scope.go:122] "RemoveContainer" containerID="c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f" Oct 30 01:09:18.578570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.578543 4517 scope.go:122] "RemoveContainer" containerID="c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f" Oct 30 01:09:18.578993 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:09:18.578964 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f\": container with ID starting with c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f not found: ID does not exist" containerID="c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f" Oct 30 01:09:18.579094 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.579016 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f"} err="failed to get container status \"c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f\": rpc error: code = NotFound desc = could not find container \"c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f\": container with ID starting with c651b2a714c218effa525bb411a0f8bad63bcf80633461d78e84de6881ca907f not found: ID does not exist" Oct 30 01:09:18.600986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.600895 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr"] Oct 30 01:09:18.610880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.610846 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd257f-86d8696c74-cbqnr"] Oct 30 01:09:18.637067 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.637025 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb"] Oct 30 01:09:18.641043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.641008 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd257f-6c8945d8d7-56lmb"] Oct 30 01:09:18.875779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.875750 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:09:18.951913 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.951879 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-plugins-config-volume\") pod \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\" (UID: \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\") " Oct 30 01:09:18.952081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.951985 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-kube-api-access-bnqfb\" (UniqueName: \"kubernetes.io/projected/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-kube-api-access-bnqfb\") pod \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\" (UID: \"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd\") " Oct 30 01:09:18.952366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.952338 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-plugins-config-volume" pod "97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" (UID: "97da81aa-246e-40ba-9d36-dbaf2f9dc9cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:18.952722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.952687 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.955058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:18.955027 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-kube-api-access-bnqfb" pod "97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" (UID: "97da81aa-246e-40ba-9d36-dbaf2f9dc9cd"). InnerVolumeSpecName "kube-api-access-bnqfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:19.054528 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.054485 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnqfb\" (UniqueName: \"kubernetes.io/projected/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd-kube-api-access-bnqfb\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:19.096541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.096442 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:19.619733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.618826 4517 generic.go:356] "Generic (PLEG): container finished" podID="97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" containerID="4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729" exitCode=0 Oct 30 01:09:19.619733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.619092 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" event={"ID":"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd","Type":"ContainerDied","Data":"4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729"} Oct 30 01:09:19.619733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.619156 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" event={"ID":"97da81aa-246e-40ba-9d36-dbaf2f9dc9cd","Type":"ContainerDied","Data":"6b11e66d5820d354a01df55d53302fe6640cd646a7a66f7542bc17beb0f00c3a"} Oct 30 01:09:19.619733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.619192 4517 scope.go:122] "RemoveContainer" containerID="4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729" Oct 30 01:09:19.619733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.619361 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn" Oct 30 01:09:19.684819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.684487 4517 scope.go:122] "RemoveContainer" containerID="4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729" Oct 30 01:09:19.685247 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:09:19.685074 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729\": container with ID starting with 4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729 not found: ID does not exist" containerID="4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729" Oct 30 01:09:19.685247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.685120 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729"} err="failed to get container status \"4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729\": rpc error: code = NotFound desc = could not find container \"4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729\": container with ID starting with 4883eec0a69f15b93430bf426bce8bb26ea067a09def76d78614423b47728729 not found: ID does not exist" Oct 30 01:09:19.760778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.756663 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn"] Oct 30 01:09:19.760778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:19.758389 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49dfaf-777485656c-7n8wn"] Oct 30 01:09:20.239166 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:20.239109 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c7e888-3215-475a-9c2e-4b4b9a9450df" path="/var/lib/kubelet/pods/35c7e888-3215-475a-9c2e-4b4b9a9450df/volumes" Oct 30 01:09:20.239977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:20.239955 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" path="/var/lib/kubelet/pods/97da81aa-246e-40ba-9d36-dbaf2f9dc9cd/volumes" Oct 30 01:09:20.240627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:20.240608 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" path="/var/lib/kubelet/pods/c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f/volumes" Oct 30 01:09:20.250631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:20.250600 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:20.468012 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:20.467982 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:21.644399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:21.644356 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:22.372618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:22.372542 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:22.488002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:22.487960 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:22.624027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:22.613795 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:22.974172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:22.974055 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:24.317863 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:24.317829 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:25.082543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:25.082506 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:25.525947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:25.525845 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:26.340393 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:26.340347 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:26.934392 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:26.934362 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:27.050130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:27.050096 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:27.278455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:27.278314 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:28.171301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:28.171246 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:29.418328 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:29.418289 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:29.775550 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:29.775434 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:29.982257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:29.982207 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:30.143736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:30.137451 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k"] Oct 30 01:09:30.143736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:30.137856 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" containerID="cri-o://7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9" gracePeriod=30 Oct 30 01:09:30.209524 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:30.209489 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:30.603989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:30.603946 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:30.762834 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:09:30.762791 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:09:32.261699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:32.261661 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:32.789293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:32.789249 4517 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:09:32.789576 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:32.789326 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:09:33.695522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:33.695453 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:34.410030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:34.409989 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:34.602160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:34.602100 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:34.717588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:34.717496 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:35.105376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.105314 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:35.534114 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.530962 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:09:35.623543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.619277 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-podinfo\") pod \"032e0c30-da56-4326-b70e-1a9ffe43554f\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " Oct 30 01:09:35.623543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.619369 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-certs\") pod \"032e0c30-da56-4326-b70e-1a9ffe43554f\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " Oct 30 01:09:35.623543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.619510 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-data\") pod \"032e0c30-da56-4326-b70e-1a9ffe43554f\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " Oct 30 01:09:35.623543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.619561 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-envoy\") pod \"032e0c30-da56-4326-b70e-1a9ffe43554f\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " Oct 30 01:09:35.623543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.619619 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-kube-api-access-cqj25\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-kube-api-access-cqj25\") pod \"032e0c30-da56-4326-b70e-1a9ffe43554f\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " Oct 30 01:09:35.623543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.619691 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-credential-socket\") pod \"032e0c30-da56-4326-b70e-1a9ffe43554f\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " Oct 30 01:09:35.623543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.619745 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032e0c30-da56-4326-b70e-1a9ffe43554f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032e0c30-da56-4326-b70e-1a9ffe43554f-istiod-ca-cert\") pod \"032e0c30-da56-4326-b70e-1a9ffe43554f\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " Oct 30 01:09:35.623543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.619794 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-socket\") pod \"032e0c30-da56-4326-b70e-1a9ffe43554f\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " Oct 30 01:09:35.623543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.620017 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-token\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-token\") pod \"032e0c30-da56-4326-b70e-1a9ffe43554f\" (UID: \"032e0c30-da56-4326-b70e-1a9ffe43554f\") " Oct 30 01:09:35.624186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.624089 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-certs" pod "032e0c30-da56-4326-b70e-1a9ffe43554f" (UID: "032e0c30-da56-4326-b70e-1a9ffe43554f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.628925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.624328 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-data" pod "032e0c30-da56-4326-b70e-1a9ffe43554f" (UID: "032e0c30-da56-4326-b70e-1a9ffe43554f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.628925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.624511 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-socket" pod "032e0c30-da56-4326-b70e-1a9ffe43554f" (UID: "032e0c30-da56-4326-b70e-1a9ffe43554f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.628925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.624594 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-credential-socket" pod "032e0c30-da56-4326-b70e-1a9ffe43554f" (UID: "032e0c30-da56-4326-b70e-1a9ffe43554f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.628925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.624621 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-token" pod "032e0c30-da56-4326-b70e-1a9ffe43554f" (UID: "032e0c30-da56-4326-b70e-1a9ffe43554f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:35.628925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.624906 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032e0c30-da56-4326-b70e-1a9ffe43554f-istiod-ca-cert" pod "032e0c30-da56-4326-b70e-1a9ffe43554f" (UID: "032e0c30-da56-4326-b70e-1a9ffe43554f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:35.640503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.640466 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-kube-api-access-cqj25" pod "032e0c30-da56-4326-b70e-1a9ffe43554f" (UID: "032e0c30-da56-4326-b70e-1a9ffe43554f"). InnerVolumeSpecName "kube-api-access-cqj25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:35.645286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.645225 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-envoy" pod "032e0c30-da56-4326-b70e-1a9ffe43554f" (UID: "032e0c30-da56-4326-b70e-1a9ffe43554f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.647801 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.647575 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-podinfo" pod "032e0c30-da56-4326-b70e-1a9ffe43554f" (UID: "032e0c30-da56-4326-b70e-1a9ffe43554f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:35.722078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.722024 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.722078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.722073 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.722078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.722083 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqj25\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-kube-api-access-cqj25\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.722078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.722093 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.722078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.722103 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032e0c30-da56-4326-b70e-1a9ffe43554f-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.722746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.722111 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.722746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.722119 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.722746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.722129 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032e0c30-da56-4326-b70e-1a9ffe43554f-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.722746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.722137 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032e0c30-da56-4326-b70e-1a9ffe43554f-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.841954 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.841913 4517 generic.go:356] "Generic (PLEG): container finished" podID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerID="7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9" exitCode=0 Oct 30 01:09:35.842133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.841961 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" Oct 30 01:09:35.842133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.841999 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" event={"ID":"032e0c30-da56-4326-b70e-1a9ffe43554f","Type":"ContainerDied","Data":"7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9"} Oct 30 01:09:35.842133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.842060 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k" event={"ID":"032e0c30-da56-4326-b70e-1a9ffe43554f","Type":"ContainerDied","Data":"09c2371243d461288c59a16408cddb88b7d8f14794577cb0f1e94d1d86a37580"} Oct 30 01:09:35.842133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.842092 4517 scope.go:122] "RemoveContainer" containerID="7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9" Oct 30 01:09:35.857957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.857932 4517 scope.go:122] "RemoveContainer" containerID="7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9" Oct 30 01:09:35.858388 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:09:35.858358 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9\": container with ID starting with 7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9 not found: ID does not exist" containerID="7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9" Oct 30 01:09:35.858493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.858398 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9"} err="failed to get container status \"7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9\": rpc error: code = NotFound desc = could not find container \"7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9\": container with ID starting with 7075c60dc97f8ec0807d94f18aa62fbb8c54a044ae611fe70ef7949d1841f5f9 not found: ID does not exist" Oct 30 01:09:35.920302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.920191 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k"] Oct 30 01:09:35.990974 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:35.990928 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-kbn5k"] Oct 30 01:09:36.237488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:36.237405 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" path="/var/lib/kubelet/pods/032e0c30-da56-4326-b70e-1a9ffe43554f/volumes" Oct 30 01:09:36.741734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:36.740114 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:38.434410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:38.434365 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:38.565528 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:38.565492 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:39.596278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:39.596246 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:41.218619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:41.218585 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:41.344731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:41.342574 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:41.573313 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:41.573267 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx"] Oct 30 01:09:41.573562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:41.573531 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" containerID="cri-o://a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb" gracePeriod=30 Oct 30 01:09:41.579880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:41.577179 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:41.978987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:41.978873 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv"] Oct 30 01:09:42.023259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:42.023217 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:43.556555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:43.556521 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:45.425376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:45.425337 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:46.911358 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.911324 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:09:46.989941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.989897 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-token\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-token\") pod \"16daf69b-dcf3-4a8e-9090-640ff89595f0\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " Oct 30 01:09:46.990164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.989981 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-credential-socket\") pod \"16daf69b-dcf3-4a8e-9090-640ff89595f0\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " Oct 30 01:09:46.990164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.990066 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16daf69b-dcf3-4a8e-9090-640ff89595f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16daf69b-dcf3-4a8e-9090-640ff89595f0-istiod-ca-cert\") pod \"16daf69b-dcf3-4a8e-9090-640ff89595f0\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " Oct 30 01:09:46.990164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.990121 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-envoy\") pod \"16daf69b-dcf3-4a8e-9090-640ff89595f0\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " Oct 30 01:09:46.990356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.990233 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-podinfo\") pod \"16daf69b-dcf3-4a8e-9090-640ff89595f0\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " Oct 30 01:09:46.990356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.990262 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-kube-api-access-4vtn7\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-kube-api-access-4vtn7\") pod \"16daf69b-dcf3-4a8e-9090-640ff89595f0\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " Oct 30 01:09:46.990356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.990302 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-data\") pod \"16daf69b-dcf3-4a8e-9090-640ff89595f0\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " Oct 30 01:09:46.990356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.990348 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-socket\") pod \"16daf69b-dcf3-4a8e-9090-640ff89595f0\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " Oct 30 01:09:46.990593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.990378 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-certs\") pod \"16daf69b-dcf3-4a8e-9090-640ff89595f0\" (UID: \"16daf69b-dcf3-4a8e-9090-640ff89595f0\") " Oct 30 01:09:46.991179 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.991155 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-certs" pod "16daf69b-dcf3-4a8e-9090-640ff89595f0" (UID: "16daf69b-dcf3-4a8e-9090-640ff89595f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:46.992611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.992018 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-socket" pod "16daf69b-dcf3-4a8e-9090-640ff89595f0" (UID: "16daf69b-dcf3-4a8e-9090-640ff89595f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:46.992611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.992381 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16daf69b-dcf3-4a8e-9090-640ff89595f0-istiod-ca-cert" pod "16daf69b-dcf3-4a8e-9090-640ff89595f0" (UID: "16daf69b-dcf3-4a8e-9090-640ff89595f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:46.992611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.992570 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-credential-socket" pod "16daf69b-dcf3-4a8e-9090-640ff89595f0" (UID: "16daf69b-dcf3-4a8e-9090-640ff89595f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:46.993149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.993128 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-data" pod "16daf69b-dcf3-4a8e-9090-640ff89595f0" (UID: "16daf69b-dcf3-4a8e-9090-640ff89595f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:46.995221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.995174 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-podinfo" pod "16daf69b-dcf3-4a8e-9090-640ff89595f0" (UID: "16daf69b-dcf3-4a8e-9090-640ff89595f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:46.998316 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.998257 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-kube-api-access-4vtn7" pod "16daf69b-dcf3-4a8e-9090-640ff89595f0" (UID: "16daf69b-dcf3-4a8e-9090-640ff89595f0"). InnerVolumeSpecName "kube-api-access-4vtn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:46.998786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.998559 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-envoy" pod "16daf69b-dcf3-4a8e-9090-640ff89595f0" (UID: "16daf69b-dcf3-4a8e-9090-640ff89595f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:46.999830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:46.999656 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-token" pod "16daf69b-dcf3-4a8e-9090-640ff89595f0" (UID: "16daf69b-dcf3-4a8e-9090-640ff89595f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:47.084096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.084053 4517 generic.go:356] "Generic (PLEG): container finished" podID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerID="a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb" exitCode=0 Oct 30 01:09:47.084287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.084112 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" event={"ID":"16daf69b-dcf3-4a8e-9090-640ff89595f0","Type":"ContainerDied","Data":"a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb"} Oct 30 01:09:47.084287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.084119 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" Oct 30 01:09:47.084287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.084169 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx" event={"ID":"16daf69b-dcf3-4a8e-9090-640ff89595f0","Type":"ContainerDied","Data":"095d71c5ba0e7d3f1d43328e685404368a3ddde81ad23b3a7f2a36342ac4007f"} Oct 30 01:09:47.084287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.084201 4517 scope.go:122] "RemoveContainer" containerID="a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb" Oct 30 01:09:47.092001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.091957 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vtn7\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-kube-api-access-4vtn7\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:47.092001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.091995 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:47.092221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.092011 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:47.092221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.092024 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:47.092221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.092036 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:47.092221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.092047 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:47.092221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.092061 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16daf69b-dcf3-4a8e-9090-640ff89595f0-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:47.092221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.092073 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:47.092221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.092084 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16daf69b-dcf3-4a8e-9090-640ff89595f0-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:47.098561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.098538 4517 scope.go:122] "RemoveContainer" containerID="a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb" Oct 30 01:09:47.099029 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:09:47.098997 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb\": container with ID starting with a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb not found: ID does not exist" containerID="a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb" Oct 30 01:09:47.099153 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.099040 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb"} err="failed to get container status \"a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb\": rpc error: code = NotFound desc = could not find container \"a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb\": container with ID starting with a1f6d7084b53952d6d3e2e659198ea48998f1b659fa3a3b238986362d9c78dcb not found: ID does not exist" Oct 30 01:09:47.132978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.132860 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx"] Oct 30 01:09:47.137078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.137035 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-w5chx"] Oct 30 01:09:47.249256 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:47.249217 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:48.237813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:48.237774 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" path="/var/lib/kubelet/pods/16daf69b-dcf3-4a8e-9090-640ff89595f0/volumes" Oct 30 01:09:48.876032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:48.875979 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:50.552392 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:50.552339 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:50.765999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:50.765954 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:50.963165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:50.963039 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:51.506022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:51.505975 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:51.724469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:51.724431 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:52.490614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:52.490565 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr"] Oct 30 01:09:52.650178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:52.650142 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:52.761233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:52.761142 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:53.403665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:53.403616 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt"] Oct 30 01:09:53.529271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:53.529215 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:53.651242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:53.651201 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:54.487487 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:54.487448 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:54.836263 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:54.836220 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:55.118886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:55.118789 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:55.244518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:55.244477 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:56.752202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:56.752161 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:56.866649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:56.866609 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:57.017927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.017804 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" containerID="cri-o://edd5543c64c99a31273e285cd594028b211a7c0c3e74fd954237aafb602ebb77" gracePeriod=45 Oct 30 01:09:57.242234 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.242171 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": dial tcp 10.133.0.141:8000: connect: connection refused" Oct 30 01:09:57.253226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.252575 4517 generic.go:356] "Generic (PLEG): container finished" podID="fb17712e-f20a-4522-9f77-0af54460a715" containerID="edd5543c64c99a31273e285cd594028b211a7c0c3e74fd954237aafb602ebb77" exitCode=0 Oct 30 01:09:57.253226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.252645 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" event={"ID":"fb17712e-f20a-4522-9f77-0af54460a715","Type":"ContainerDied","Data":"edd5543c64c99a31273e285cd594028b211a7c0c3e74fd954237aafb602ebb77"} Oct 30 01:09:57.439098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.439065 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:09:57.491541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.491493 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" podUID="9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" containerName="payload-processing" containerID="cri-o://48547028e1480cbc8b496cde7522f32e6f1513cf295077839726f9516c0fb6aa" gracePeriod=25 Oct 30 01:09:57.546691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.546654 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-home\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-home\") pod \"fb17712e-f20a-4522-9f77-0af54460a715\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " Oct 30 01:09:57.546932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.546742 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-dshm\") pod \"fb17712e-f20a-4522-9f77-0af54460a715\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " Oct 30 01:09:57.546932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.546856 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb17712e-f20a-4522-9f77-0af54460a715-kube-api-access-kbz5l\" (UniqueName: \"kubernetes.io/projected/fb17712e-f20a-4522-9f77-0af54460a715-kube-api-access-kbz5l\") pod \"fb17712e-f20a-4522-9f77-0af54460a715\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " Oct 30 01:09:57.546932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.546884 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-model-cache\") pod \"fb17712e-f20a-4522-9f77-0af54460a715\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " Oct 30 01:09:57.546932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.546914 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb17712e-f20a-4522-9f77-0af54460a715-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb17712e-f20a-4522-9f77-0af54460a715-tls-certs\") pod \"fb17712e-f20a-4522-9f77-0af54460a715\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " Oct 30 01:09:57.547210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.547001 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-tmp-dir\") pod \"fb17712e-f20a-4522-9f77-0af54460a715\" (UID: \"fb17712e-f20a-4522-9f77-0af54460a715\") " Oct 30 01:09:57.547210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.547103 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-home" pod "fb17712e-f20a-4522-9f77-0af54460a715" (UID: "fb17712e-f20a-4522-9f77-0af54460a715"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:57.547323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.547294 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-tmp-dir" pod "fb17712e-f20a-4522-9f77-0af54460a715" (UID: "fb17712e-f20a-4522-9f77-0af54460a715"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:57.547557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.547496 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-model-cache" pod "fb17712e-f20a-4522-9f77-0af54460a715" (UID: "fb17712e-f20a-4522-9f77-0af54460a715"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:57.547789 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.547762 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:57.547789 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.547791 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:57.547957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.547806 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:57.549507 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.549485 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-dshm" pod "fb17712e-f20a-4522-9f77-0af54460a715" (UID: "fb17712e-f20a-4522-9f77-0af54460a715"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:57.549675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.549656 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb17712e-f20a-4522-9f77-0af54460a715-tls-certs" pod "fb17712e-f20a-4522-9f77-0af54460a715" (UID: "fb17712e-f20a-4522-9f77-0af54460a715"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:57.549819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.549796 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb17712e-f20a-4522-9f77-0af54460a715-kube-api-access-kbz5l" pod "fb17712e-f20a-4522-9f77-0af54460a715" (UID: "fb17712e-f20a-4522-9f77-0af54460a715"). InnerVolumeSpecName "kube-api-access-kbz5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:57.649346 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.649303 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb17712e-f20a-4522-9f77-0af54460a715-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:57.649346 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.649339 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbz5l\" (UniqueName: \"kubernetes.io/projected/fb17712e-f20a-4522-9f77-0af54460a715-kube-api-access-kbz5l\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:57.649346 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:57.649348 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb17712e-f20a-4522-9f77-0af54460a715-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:09:58.264036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:58.263976 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" event={"ID":"fb17712e-f20a-4522-9f77-0af54460a715","Type":"ContainerDied","Data":"bea01db453414441da06c96fb59a4924f1ff25504e3e4f423cb7dd9234aa14e4"} Oct 30 01:09:58.264036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:58.264035 4517 scope.go:122] "RemoveContainer" containerID="edd5543c64c99a31273e285cd594028b211a7c0c3e74fd954237aafb602ebb77" Oct 30 01:09:58.264589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:58.263985 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv" Oct 30 01:09:58.312929 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:58.312886 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv"] Oct 30 01:09:58.318465 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:58.318427 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f96ede/test-model-f96ede-kserve-77bb6d6497-qlsnv"] Oct 30 01:09:58.405188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:58.405116 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" podUID="1503f898-c417-4ef8-8a3f-a33eed038eb6" containerName="payload-pre-processing" containerID="cri-o://62b02136c4875fa73693495942e6371bb33c0b105adabede63e103c52ea45663" gracePeriod=25 Oct 30 01:09:58.847399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:58.847367 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:59.745066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:09:59.745013 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:00.236967 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:00.236922 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb17712e-f20a-4522-9f77-0af54460a715" path="/var/lib/kubelet/pods/fb17712e-f20a-4522-9f77-0af54460a715/volumes" Oct 30 01:10:00.774264 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:10:00.774220 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:10:00.876592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:00.876555 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:02.183262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.183217 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:02.307853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.307809 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv"] Oct 30 01:10:02.308514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308492 4517 cpu_manager.go:414] "containerMap: removing container" podUID="97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" containerName="payload-pre-processing" Oct 30 01:10:02.308514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308512 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" containerName="payload-pre-processing" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308535 4517 cpu_manager.go:414] "containerMap: removing container" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308541 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308553 4517 cpu_manager.go:414] "containerMap: removing container" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308563 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308591 4517 cpu_manager.go:414] "containerMap: removing container" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308595 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308602 4517 cpu_manager.go:414] "containerMap: removing container" podUID="35c7e888-3215-475a-9c2e-4b4b9a9450df" containerName="payload-pre-processing" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308607 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c7e888-3215-475a-9c2e-4b4b9a9450df" containerName="payload-pre-processing" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308618 4517 cpu_manager.go:414] "containerMap: removing container" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308623 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308640 4517 cpu_manager.go:414] "containerMap: removing container" podUID="c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" containerName="payload-processing" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308650 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" containerName="payload-processing" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308661 4517 cpu_manager.go:414] "containerMap: removing container" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" Oct 30 01:10:02.308763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308666 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" Oct 30 01:10:02.309463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308788 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="16daf69b-dcf3-4a8e-9090-640ff89595f0" containerName="istio-proxy" Oct 30 01:10:02.309463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308803 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c7e888-3215-475a-9c2e-4b4b9a9450df" containerName="payload-pre-processing" Oct 30 01:10:02.309463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308817 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="032e0c30-da56-4326-b70e-1a9ffe43554f" containerName="istio-proxy" Oct 30 01:10:02.309463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308826 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="c486bf7f-4d59-4ab1-9e75-ff2ce2b13b5f" containerName="payload-processing" Oct 30 01:10:02.309463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308832 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd017f4b-0b2f-409b-a895-79c151564111" containerName="main" Oct 30 01:10:02.309463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308837 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="97da81aa-246e-40ba-9d36-dbaf2f9dc9cd" containerName="payload-pre-processing" Oct 30 01:10:02.309463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308844 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8a756b-d0ec-4fec-952c-4ae5e28b54f8" containerName="istio-proxy" Oct 30 01:10:02.309463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.308850 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb17712e-f20a-4522-9f77-0af54460a715" containerName="main" Oct 30 01:10:02.314263 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.314228 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.337013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.336971 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv"] Oct 30 01:10:02.341049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.341017 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1da7b2-openshift-default-dockercfg-zwj8q\"" Oct 30 01:10:02.491451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.483550 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-envoy\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.491451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.483633 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-podinfo\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.491451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.483789 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-credential-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.491451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.483820 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-certs\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.491451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.483856 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istiod-ca-cert\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.491451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.483919 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-data\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.491451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.483947 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54mj\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-kube-api-access-h54mj\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.491451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.483993 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.491451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.484022 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-token\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.585887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.585385 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-envoy\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.585887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.585453 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-podinfo\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.585887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.585576 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-credential-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.585887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.585605 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-certs\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.585887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.585641 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istiod-ca-cert\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.585887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.585694 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-data\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.585887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.585740 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54mj\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-kube-api-access-h54mj\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.585887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.585783 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.586535 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.586311 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-certs\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.587390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.586618 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-token\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.587996 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.587973 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istiod-ca-cert\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.588422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.588395 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.588917 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.588532 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-data\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.589095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.588828 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-credential-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.595680 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.595647 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-envoy\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.614607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.614465 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-podinfo\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.617435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.617390 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-token\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.620151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.619696 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54mj\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-kube-api-access-h54mj\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:02.632776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.632103 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:02.636585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:02.636548 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:03.058237 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:03.058206 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv"] Oct 30 01:10:03.060840 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:10:03.060806 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d4bf7c_8e15_4e8c_9be3_8276d63e9ac8.slice/crio-2414ac6488f2719c55ab446c6b25d88b6c011ef67e0d16dcd0ec16452b477c0e WatchSource:0}: Error finding container 2414ac6488f2719c55ab446c6b25d88b6c011ef67e0d16dcd0ec16452b477c0e: Status 404 returned error can't find the container with id 2414ac6488f2719c55ab446c6b25d88b6c011ef67e0d16dcd0ec16452b477c0e Oct 30 01:10:03.063950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:03.063909 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:03.064074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:03.063983 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:03.064074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:03.064015 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:03.309849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:03.309696 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" event={"ID":"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8","Type":"ContainerStarted","Data":"4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3"} Oct 30 01:10:03.309849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:03.309814 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" event={"ID":"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8","Type":"ContainerStarted","Data":"2414ac6488f2719c55ab446c6b25d88b6c011ef67e0d16dcd0ec16452b477c0e"} Oct 30 01:10:03.355787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:03.355690 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" podStartSLOduration=1.355664857 podStartE2EDuration="1.355664857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:03.351231263 +0000 UTC m=+2141.263358405" watchObservedRunningTime="2026-10-30 01:10:03.355664857 +0000 UTC m=+2141.267791998" Oct 30 01:10:03.638165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:03.638066 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:04.199731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:04.198995 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:04.641756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:04.641131 4517 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:10:04.641756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:04.641202 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 01:10:05.029910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:05.029822 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:05.171736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:05.166348 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:05.287075 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:05.287035 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:05.587466 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:05.587323 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" podUID="1503f898-c417-4ef8-8a3f-a33eed038eb6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:05.603953 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:05.603917 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:05.638052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:05.638013 4517 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:10:05.638280 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:05.638104 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 01:10:06.638242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:06.638202 4517 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:10:06.638919 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:06.638882 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 01:10:06.705208 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:06.705165 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:07.327515 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:07.327431 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:07.327968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:07.327875 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" podUID="9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:07.449572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:07.449521 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:07.592802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:07.580325 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:07.639240 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:07.638743 4517 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:10:07.639240 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:07.638825 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 01:10:07.911732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:07.911620 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:08.638336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.638277 4517 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:10:08.638584 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.638354 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 01:10:08.898375 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.898277 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h"] Oct 30 01:10:08.904214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.904186 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:08.934948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.934878 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h"] Oct 30 01:10:08.990224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.990183 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-16c1bb-openshift-default-dockercfg-h52zt\"" Oct 30 01:10:08.997436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.997392 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:08.997620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.997551 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:08.997620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.997591 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:08.997695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.997625 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251c373b-a73a-4aef-b678-b579391a6b96-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:08.997768 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.997723 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:08.997818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.997801 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6rds\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-kube-api-access-v6rds\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:08.997858 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.997842 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:08.997892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.997882 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251c373b-a73a-4aef-b678-b579391a6b96-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:08.997938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:08.997925 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.100049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.099991 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.100258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.100070 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.100258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.100105 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251c373b-a73a-4aef-b678-b579391a6b96-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.100258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.100182 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.100378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.100260 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6rds\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-kube-api-access-v6rds\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.100378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.100297 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.100378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.100338 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251c373b-a73a-4aef-b678-b579391a6b96-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.100474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.100384 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.100530 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.100511 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.101076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.101049 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.101275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.101106 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.101275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.101238 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.101937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.101912 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251c373b-a73a-4aef-b678-b579391a6b96-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.102238 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.102195 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.104045 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.103973 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.104395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.104372 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251c373b-a73a-4aef-b678-b579391a6b96-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.123323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.123287 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.123558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.123465 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6rds\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-kube-api-access-v6rds\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.226545 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.226444 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:09.333526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.333342 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:09.594663 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.594611 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h"] Oct 30 01:10:09.596767 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:10:09.596725 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251c373b_a73a_4aef_b678_b579391a6b96.slice/crio-845717606f22ad6da0a0b62a063ffe209c79fecca67257e00fb956f849a31df9 WatchSource:0}: Error finding container 845717606f22ad6da0a0b62a063ffe209c79fecca67257e00fb956f849a31df9: Status 404 returned error can't find the container with id 845717606f22ad6da0a0b62a063ffe209c79fecca67257e00fb956f849a31df9 Oct 30 01:10:09.600504 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.600454 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:09.600885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.600845 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:09.601015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.600945 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:09.637847 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.637807 4517 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:10:09.638029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:09.637873 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 01:10:10.434458 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:10.434418 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" event={"ID":"251c373b-a73a-4aef-b678-b579391a6b96","Type":"ContainerStarted","Data":"e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012"} Oct 30 01:10:10.434978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:10.434485 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" event={"ID":"251c373b-a73a-4aef-b678-b579391a6b96","Type":"ContainerStarted","Data":"845717606f22ad6da0a0b62a063ffe209c79fecca67257e00fb956f849a31df9"} Oct 30 01:10:10.490193 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:10.487774 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" podStartSLOduration=2.487755967 podStartE2EDuration="2.487755967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:10.487072398 +0000 UTC m=+2148.399199558" watchObservedRunningTime="2026-10-30 01:10:10.487755967 +0000 UTC m=+2148.399883109" Oct 30 01:10:10.642784 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:10.642745 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:10.643190 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:10.643163 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:10.643944 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:10.643926 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:11.152107 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:11.152065 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:11.233735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:11.229579 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:11.233735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:11.232898 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:10:11.233735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:11.232958 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:10:11.982633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:11.982589 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:12.120800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:12.120759 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:12.227332 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:12.227292 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:10:12.227554 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:12.227357 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:10:12.327560 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:12.327502 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:12.996735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:12.996102 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:13.118379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:13.118337 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:13.227892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:13.227856 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:10:13.228090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:13.227914 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:10:13.843791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:13.825686 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:13.984835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:13.984792 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:14.227137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:14.227032 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:10:14.227137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:14.227098 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:10:14.735009 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:14.734973 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:15.040396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:15.040361 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:15.174323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:15.174285 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:15.226987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:15.226917 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:10:15.226987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:15.226978 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:10:15.436351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:15.436245 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:15.617941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:15.617876 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" podUID="1503f898-c417-4ef8-8a3f-a33eed038eb6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:16.226922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:16.226879 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:10:16.227168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:16.226956 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:10:16.705642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:16.705602 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:16.827734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:16.824157 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:17.232592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.232560 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:17.233129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.232984 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:17.233916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.233896 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:10:17.317968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.317925 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" podUID="9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:17.562389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.562347 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:17.618439 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.618405 4517 generic.go:356] "Generic (PLEG): container finished" podID="9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" containerID="48547028e1480cbc8b496cde7522f32e6f1513cf295077839726f9516c0fb6aa" exitCode=0 Oct 30 01:10:17.618602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.618485 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" event={"ID":"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547","Type":"ContainerDied","Data":"48547028e1480cbc8b496cde7522f32e6f1513cf295077839726f9516c0fb6aa"} Oct 30 01:10:17.695435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.695158 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:17.821793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.818202 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:10:17.911424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.911383 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-kube-api-access-jh7pt\" (UniqueName: \"kubernetes.io/projected/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-kube-api-access-jh7pt\") pod \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\" (UID: \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\") " Oct 30 01:10:17.911853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.911830 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-plugins-config-volume\") pod \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\" (UID: \"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547\") " Oct 30 01:10:17.912287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.912237 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-plugins-config-volume" pod "9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" (UID: "9d693d57-a4e9-4efa-a6eb-0a19cbd6a547"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:17.913201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.913166 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:17.914436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:17.914407 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-kube-api-access-jh7pt" pod "9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" (UID: "9d693d57-a4e9-4efa-a6eb-0a19cbd6a547"). InnerVolumeSpecName "kube-api-access-jh7pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:18.015305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.015199 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh7pt\" (UniqueName: \"kubernetes.io/projected/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547-kube-api-access-jh7pt\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:18.627580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.627529 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:18.632642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.632608 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" Oct 30 01:10:18.632642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.632619 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr" event={"ID":"9d693d57-a4e9-4efa-a6eb-0a19cbd6a547","Type":"ContainerDied","Data":"31a6d490d46f7bc77608574768eb8e15f66a3ccb606394ac6e7ed43aa8c69b6e"} Oct 30 01:10:18.632987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.632694 4517 scope.go:122] "RemoveContainer" containerID="48547028e1480cbc8b496cde7522f32e6f1513cf295077839726f9516c0fb6aa" Oct 30 01:10:18.634961 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.634930 4517 generic.go:356] "Generic (PLEG): container finished" podID="1503f898-c417-4ef8-8a3f-a33eed038eb6" containerID="62b02136c4875fa73693495942e6371bb33c0b105adabede63e103c52ea45663" exitCode=0 Oct 30 01:10:18.635744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.635644 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" event={"ID":"1503f898-c417-4ef8-8a3f-a33eed038eb6","Type":"ContainerDied","Data":"62b02136c4875fa73693495942e6371bb33c0b105adabede63e103c52ea45663"} Oct 30 01:10:18.718156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.717977 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr"] Oct 30 01:10:18.718156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.718045 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf3466-5858df687b-khkpr"] Oct 30 01:10:18.796593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.796550 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:10:18.928607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.928562 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1503f898-c417-4ef8-8a3f-a33eed038eb6-kube-api-access-t2w64\" (UniqueName: \"kubernetes.io/projected/1503f898-c417-4ef8-8a3f-a33eed038eb6-kube-api-access-t2w64\") pod \"1503f898-c417-4ef8-8a3f-a33eed038eb6\" (UID: \"1503f898-c417-4ef8-8a3f-a33eed038eb6\") " Oct 30 01:10:18.928835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.928671 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1503f898-c417-4ef8-8a3f-a33eed038eb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1503f898-c417-4ef8-8a3f-a33eed038eb6-plugins-config-volume\") pod \"1503f898-c417-4ef8-8a3f-a33eed038eb6\" (UID: \"1503f898-c417-4ef8-8a3f-a33eed038eb6\") " Oct 30 01:10:18.929045 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.929015 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1503f898-c417-4ef8-8a3f-a33eed038eb6-plugins-config-volume" pod "1503f898-c417-4ef8-8a3f-a33eed038eb6" (UID: "1503f898-c417-4ef8-8a3f-a33eed038eb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:18.929324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.929301 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1503f898-c417-4ef8-8a3f-a33eed038eb6-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:18.931260 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:18.931223 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1503f898-c417-4ef8-8a3f-a33eed038eb6-kube-api-access-t2w64" pod "1503f898-c417-4ef8-8a3f-a33eed038eb6" (UID: "1503f898-c417-4ef8-8a3f-a33eed038eb6"). InnerVolumeSpecName "kube-api-access-t2w64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:19.032396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:19.032305 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2w64\" (UniqueName: \"kubernetes.io/projected/1503f898-c417-4ef8-8a3f-a33eed038eb6-kube-api-access-t2w64\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:19.632697 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:19.632658 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:19.647661 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:19.647623 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" Oct 30 01:10:19.647901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:19.647630 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt" event={"ID":"1503f898-c417-4ef8-8a3f-a33eed038eb6","Type":"ContainerDied","Data":"cf56e48eb63f36fd03b09c4f034ecf80f4a2fb42eefd00ffcdcb38588dbc7d25"} Oct 30 01:10:19.647901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:19.647824 4517 scope.go:122] "RemoveContainer" containerID="62b02136c4875fa73693495942e6371bb33c0b105adabede63e103c52ea45663" Oct 30 01:10:19.699976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:19.699930 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt"] Oct 30 01:10:19.709968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:19.709934 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1da7b2-74c4ddb675-hbkbt"] Oct 30 01:10:19.753281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:19.753227 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:20.236960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:20.236927 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1503f898-c417-4ef8-8a3f-a33eed038eb6" path="/var/lib/kubelet/pods/1503f898-c417-4ef8-8a3f-a33eed038eb6/volumes" Oct 30 01:10:20.237403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:20.237389 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" path="/var/lib/kubelet/pods/9d693d57-a4e9-4efa-a6eb-0a19cbd6a547/volumes" Oct 30 01:10:20.620718 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:20.620667 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:22.324542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:22.324506 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:23.621964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:23.621915 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf"] Oct 30 01:10:23.622541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:23.622290 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" podUID="15dc04f8-79b8-41d7-8ada-df7ed4019560" containerName="maas-api" containerID="cri-o://c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1" gracePeriod=30 Oct 30 01:10:23.704173 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:23.704092 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:24.111658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.098298 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft"] Oct 30 01:10:24.145385 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.145342 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1503f898-c417-4ef8-8a3f-a33eed038eb6" containerName="payload-pre-processing" Oct 30 01:10:24.145635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.145616 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1503f898-c417-4ef8-8a3f-a33eed038eb6" containerName="payload-pre-processing" Oct 30 01:10:24.145804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.145791 4517 cpu_manager.go:414] "containerMap: removing container" podUID="9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" containerName="payload-processing" Oct 30 01:10:24.145907 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.145896 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" containerName="payload-processing" Oct 30 01:10:24.146239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.146222 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d693d57-a4e9-4efa-a6eb-0a19cbd6a547" containerName="payload-processing" Oct 30 01:10:24.146368 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.146354 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1503f898-c417-4ef8-8a3f-a33eed038eb6" containerName="payload-pre-processing" Oct 30 01:10:24.335617 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.335568 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.418651 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.420658 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa942ab-6c15-481f-8936-6b2c21358635-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.420823 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.420974 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.421009 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.421073 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.421112 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa942ab-6c15-481f-8936-6b2c21358635-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.421180 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.421235 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.426120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.421266 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97kp2\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-kube-api-access-97kp2\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.487811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.462190 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft"] Oct 30 01:10:24.525002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.522987 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.525002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.523052 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97kp2\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-kube-api-access-97kp2\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.525002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.523165 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa942ab-6c15-481f-8936-6b2c21358635-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.525002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.523233 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.525002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.523364 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.525002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.523401 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.525002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.523465 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.525002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.523503 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa942ab-6c15-481f-8936-6b2c21358635-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.525002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.523572 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.528808 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.528773 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.536395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.535060 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.536395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.535518 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.536395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.535811 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.545879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.545828 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa942ab-6c15-481f-8936-6b2c21358635-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.561020 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.560540 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97kp2\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-kube-api-access-97kp2\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.588777 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.588742 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:10:24.590125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.590088 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:24.590368 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.590098 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.605427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.591272 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.634196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.631054 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls\") pod \"15dc04f8-79b8-41d7-8ada-df7ed4019560\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " Oct 30 01:10:24.634196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.631131 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15dc04f8-79b8-41d7-8ada-df7ed4019560-kube-api-access-q9vkz\" (UniqueName: \"kubernetes.io/projected/15dc04f8-79b8-41d7-8ada-df7ed4019560-kube-api-access-q9vkz\") pod \"15dc04f8-79b8-41d7-8ada-df7ed4019560\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " Oct 30 01:10:24.642151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.642107 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa942ab-6c15-481f-8936-6b2c21358635-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.667771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.667587 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:24.684763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.684662 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15dc04f8-79b8-41d7-8ada-df7ed4019560-kube-api-access-q9vkz" pod "15dc04f8-79b8-41d7-8ada-df7ed4019560" (UID: "15dc04f8-79b8-41d7-8ada-df7ed4019560"). InnerVolumeSpecName "kube-api-access-q9vkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:24.745297 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.745258 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:24.754178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.754119 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15dc04f8-79b8-41d7-8ada-df7ed4019560-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dc04f8-79b8-41d7-8ada-df7ed4019560-openshift-service-ca\") pod \"15dc04f8-79b8-41d7-8ada-df7ed4019560\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " Oct 30 01:10:24.763728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.763669 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls" pod "15dc04f8-79b8-41d7-8ada-df7ed4019560" (UID: "15dc04f8-79b8-41d7-8ada-df7ed4019560"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:24.774911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.768247 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15dc04f8-79b8-41d7-8ada-df7ed4019560-openshift-service-ca" pod "15dc04f8-79b8-41d7-8ada-df7ed4019560" (UID: "15dc04f8-79b8-41d7-8ada-df7ed4019560"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:24.775165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.773842 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-metrics-certs\") pod \"15dc04f8-79b8-41d7-8ada-df7ed4019560\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " Oct 30 01:10:24.775389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.775371 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls\") pod \"15dc04f8-79b8-41d7-8ada-df7ed4019560\" (UID: \"15dc04f8-79b8-41d7-8ada-df7ed4019560\") " Oct 30 01:10:24.785787 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:10:24.779202 4517 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/15dc04f8-79b8-41d7-8ada-df7ed4019560/volumes/kubernetes.io~secret/maas-api-tls Oct 30 01:10:24.785787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.779231 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls" pod "15dc04f8-79b8-41d7-8ada-df7ed4019560" (UID: "15dc04f8-79b8-41d7-8ada-df7ed4019560"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:24.785787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.779770 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:24.785787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.779804 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9vkz\" (UniqueName: \"kubernetes.io/projected/15dc04f8-79b8-41d7-8ada-df7ed4019560-kube-api-access-q9vkz\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:24.785787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.779826 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15dc04f8-79b8-41d7-8ada-df7ed4019560-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:24.789989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.789478 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-metrics-certs" pod "15dc04f8-79b8-41d7-8ada-df7ed4019560" (UID: "15dc04f8-79b8-41d7-8ada-df7ed4019560"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:24.894218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.889545 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15dc04f8-79b8-41d7-8ada-df7ed4019560-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:24.921027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.915537 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" Oct 30 01:10:24.921027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.915446 4517 generic.go:356] "Generic (PLEG): container finished" podID="15dc04f8-79b8-41d7-8ada-df7ed4019560" containerID="c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1" exitCode=0 Oct 30 01:10:24.923769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.921131 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" event={"ID":"15dc04f8-79b8-41d7-8ada-df7ed4019560","Type":"ContainerDied","Data":"c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1"} Oct 30 01:10:24.923769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.921206 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf" event={"ID":"15dc04f8-79b8-41d7-8ada-df7ed4019560","Type":"ContainerDied","Data":"8c40f15450da7fc27dcf10cb5b93793d6592e5469e5329c7e3d472565882329c"} Oct 30 01:10:24.923769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.921243 4517 scope.go:122] "RemoveContainer" containerID="c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1" Oct 30 01:10:24.937165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.937098 4517 scope.go:122] "RemoveContainer" containerID="c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1" Oct 30 01:10:24.937528 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:10:24.937500 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1\": container with ID starting with c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1 not found: ID does not exist" containerID="c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1" Oct 30 01:10:24.937694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.937668 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1"} err="failed to get container status \"c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1\": rpc error: code = NotFound desc = could not find container \"c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1\": container with ID starting with c9f5bd98f4d3fa93a08c6c566eafe146b504070b7afa102cad38b69a74be48d1 not found: ID does not exist" Oct 30 01:10:24.942197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.942166 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:24.982023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.971870 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf"] Oct 30 01:10:24.993786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:24.991636 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1da7b2-5cbb66c479-4pxhf"] Oct 30 01:10:25.308049 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:10:25.308010 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa942ab_6c15_481f_8936_6b2c21358635.slice/crio-61e7f369ed3ad8b64ac95f5f51a172299a20b3f4a4fadd1b45add2888f9e2ce0 WatchSource:0}: Error finding container 61e7f369ed3ad8b64ac95f5f51a172299a20b3f4a4fadd1b45add2888f9e2ce0: Status 404 returned error can't find the container with id 61e7f369ed3ad8b64ac95f5f51a172299a20b3f4a4fadd1b45add2888f9e2ce0 Oct 30 01:10:25.310458 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:25.310427 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft"] Oct 30 01:10:25.311466 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:25.311427 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:25.311603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:25.311515 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:25.311603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:25.311563 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:25.932188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:25.932141 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" event={"ID":"1fa942ab-6c15-481f-8936-6b2c21358635","Type":"ContainerStarted","Data":"ff36c1db0433d17a419d7209c5227f0d37bd089d89506bd278ae046006299a9e"} Oct 30 01:10:25.932650 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:25.932250 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" event={"ID":"1fa942ab-6c15-481f-8936-6b2c21358635","Type":"ContainerStarted","Data":"61e7f369ed3ad8b64ac95f5f51a172299a20b3f4a4fadd1b45add2888f9e2ce0"} Oct 30 01:10:25.987786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:25.987698 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" podStartSLOduration=1.987672216 podStartE2EDuration="1.987672216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:25.983365994 +0000 UTC m=+2163.895493153" watchObservedRunningTime="2026-10-30 01:10:25.987672216 +0000 UTC m=+2163.899799361" Oct 30 01:10:26.237358 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:26.237272 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15dc04f8-79b8-41d7-8ada-df7ed4019560" path="/var/lib/kubelet/pods/15dc04f8-79b8-41d7-8ada-df7ed4019560/volumes" Oct 30 01:10:26.302214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:26.302174 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk"] Oct 30 01:10:26.330049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:26.330009 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r"] Oct 30 01:10:26.509856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:26.509776 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:26.628477 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:26.628423 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:26.694742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:26.692275 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:26.710447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:26.710386 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:10:26.711088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:26.711037 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:10:27.690232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:27.690192 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:10:27.690899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:27.690865 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:10:28.686672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:28.686172 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:10:28.686672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:28.686260 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:10:28.891307 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:28.891250 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:29.684219 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:29.684168 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:10:29.684581 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:29.684550 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:10:30.657169 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:30.657136 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:30.686218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:30.686175 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:10:30.694334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:30.686509 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:10:30.781259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:30.781175 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:30.796398 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:10:30.796354 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:10:30.905735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:30.904947 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:31.186732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:31.186308 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:31.302950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:31.302899 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" podUID="d34572ac-cb49-4dfd-b400-248d93f99f68" containerName="payload-processing" containerID="cri-o://f23479882d2939b589a3309d3e58fe9378205d72e2ea8a4ea996b9238f19e87f" gracePeriod=25 Oct 30 01:10:31.330809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:31.330773 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" podUID="25469977-6f4f-4788-ad6a-1f6f91c84ccf" containerName="payload-pre-processing" containerID="cri-o://d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582" gracePeriod=25 Oct 30 01:10:31.438011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:31.437873 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:31.627384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:31.627347 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:31.634049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:31.633657 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv"] Oct 30 01:10:31.634049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:31.633977 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" containerID="cri-o://4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3" gracePeriod=30 Oct 30 01:10:31.708018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:31.707929 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:10:31.715371 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:31.714852 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:10:32.107290 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.107255 4517 generic.go:356] "Generic (PLEG): container finished" podID="d34572ac-cb49-4dfd-b400-248d93f99f68" containerID="f23479882d2939b589a3309d3e58fe9378205d72e2ea8a4ea996b9238f19e87f" exitCode=0 Oct 30 01:10:32.107680 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.107336 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" event={"ID":"d34572ac-cb49-4dfd-b400-248d93f99f68","Type":"ContainerDied","Data":"f23479882d2939b589a3309d3e58fe9378205d72e2ea8a4ea996b9238f19e87f"} Oct 30 01:10:32.109039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.108217 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:10:32.211627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.202561 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34572ac-cb49-4dfd-b400-248d93f99f68-kube-api-access-p4tcc\" (UniqueName: \"kubernetes.io/projected/d34572ac-cb49-4dfd-b400-248d93f99f68-kube-api-access-p4tcc\") pod \"d34572ac-cb49-4dfd-b400-248d93f99f68\" (UID: \"d34572ac-cb49-4dfd-b400-248d93f99f68\") " Oct 30 01:10:32.211627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.210491 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34572ac-cb49-4dfd-b400-248d93f99f68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34572ac-cb49-4dfd-b400-248d93f99f68-plugins-config-volume\") pod \"d34572ac-cb49-4dfd-b400-248d93f99f68\" (UID: \"d34572ac-cb49-4dfd-b400-248d93f99f68\") " Oct 30 01:10:32.215401 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.215358 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34572ac-cb49-4dfd-b400-248d93f99f68-plugins-config-volume" pod "d34572ac-cb49-4dfd-b400-248d93f99f68" (UID: "d34572ac-cb49-4dfd-b400-248d93f99f68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:32.222595 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.222397 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34572ac-cb49-4dfd-b400-248d93f99f68-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:32.224564 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.224523 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34572ac-cb49-4dfd-b400-248d93f99f68-kube-api-access-p4tcc" pod "d34572ac-cb49-4dfd-b400-248d93f99f68" (UID: "d34572ac-cb49-4dfd-b400-248d93f99f68"). InnerVolumeSpecName "kube-api-access-p4tcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:32.349272 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.330496 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tcc\" (UniqueName: \"kubernetes.io/projected/d34572ac-cb49-4dfd-b400-248d93f99f68-kube-api-access-p4tcc\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:32.684042 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.683947 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:10:32.684236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.684030 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:10:32.986848 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:32.986750 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:33.117240 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:33.117211 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" Oct 30 01:10:33.117403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:33.117251 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk" event={"ID":"d34572ac-cb49-4dfd-b400-248d93f99f68","Type":"ContainerDied","Data":"e690dbb9f0236b15f4d81361670c9e98f4842b756eebb3581a204ab67119de33"} Oct 30 01:10:33.117403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:33.117339 4517 scope.go:122] "RemoveContainer" containerID="f23479882d2939b589a3309d3e58fe9378205d72e2ea8a4ea996b9238f19e87f" Oct 30 01:10:33.172494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:33.172453 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk"] Oct 30 01:10:33.184911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:33.184870 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f96ede-55bbbb7db8-956nk"] Oct 30 01:10:33.753600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:33.753562 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:33.754528 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:33.754495 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:33.756542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:33.756518 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:10:34.285732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:34.282813 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34572ac-cb49-4dfd-b400-248d93f99f68" path="/var/lib/kubelet/pods/d34572ac-cb49-4dfd-b400-248d93f99f68/volumes" Oct 30 01:10:35.151424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:35.151383 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:36.972884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:36.953121 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:37.061691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.061662 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:37.121618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.121156 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-podinfo\") pod \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " Oct 30 01:10:37.122096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.122067 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-kube-api-access-h54mj\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-kube-api-access-h54mj\") pod \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " Oct 30 01:10:37.122766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.122744 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istiod-ca-cert\") pod \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " Oct 30 01:10:37.122994 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.122974 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-socket\") pod \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " Oct 30 01:10:37.123159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.123139 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-credential-socket\") pod \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " Oct 30 01:10:37.123366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.123349 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-data\") pod \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " Oct 30 01:10:37.123598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.123581 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-certs\") pod \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " Oct 30 01:10:37.123895 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.123875 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-envoy\") pod \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " Oct 30 01:10:37.124043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.124027 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-token\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-token\") pod \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\" (UID: \"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8\") " Oct 30 01:10:37.125642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.125453 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-credential-socket" pod "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" (UID: "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.126155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.125916 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istiod-ca-cert" pod "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" (UID: "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:37.126474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.126296 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-socket" pod "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" (UID: "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.126790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.126600 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-certs" pod "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" (UID: "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.127669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.126934 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-data" pod "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" (UID: "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.134117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.134079 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-envoy" pod "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" (UID: "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.134369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.134224 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-kube-api-access-h54mj" pod "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" (UID: "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8"). InnerVolumeSpecName "kube-api-access-h54mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:37.134369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.134254 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-token" pod "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" (UID: "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:37.134859 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.134835 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-podinfo" pod "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" (UID: "41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:10:37.154378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.154340 4517 generic.go:356] "Generic (PLEG): container finished" podID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerID="4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3" exitCode=0 Oct 30 01:10:37.154604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.154392 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" Oct 30 01:10:37.154604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.154491 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" event={"ID":"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8","Type":"ContainerDied","Data":"4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3"} Oct 30 01:10:37.154604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.154536 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv" event={"ID":"41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8","Type":"ContainerDied","Data":"2414ac6488f2719c55ab446c6b25d88b6c011ef67e0d16dcd0ec16452b477c0e"} Oct 30 01:10:37.154604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.154564 4517 scope.go:122] "RemoveContainer" containerID="4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3" Oct 30 01:10:37.171420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.171393 4517 scope.go:122] "RemoveContainer" containerID="4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3" Oct 30 01:10:37.171893 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:10:37.171868 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3\": container with ID starting with 4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3 not found: ID does not exist" containerID="4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3" Oct 30 01:10:37.171991 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.171903 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3"} err="failed to get container status \"4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3\": rpc error: code = NotFound desc = could not find container \"4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3\": container with ID starting with 4c3d94c148a154d069502dbcc2a8142fad7b8b1f3794a6f4bb40e84bddf4eeb3 not found: ID does not exist" Oct 30 01:10:37.204857 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.204770 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv"] Oct 30 01:10:37.208699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.208662 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-5z9dv"] Oct 30 01:10:37.226989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.226887 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h54mj\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-kube-api-access-h54mj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.226989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.226927 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.226989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.226941 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.226989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.226952 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.226989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.226965 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.226989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.226976 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.226989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.226988 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.227374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.227002 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.227374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:37.227013 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:38.239132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.239097 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" path="/var/lib/kubelet/pods/41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8/volumes" Oct 30 01:10:38.354494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.354457 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" podUID="25469977-6f4f-4788-ad6a-1f6f91c84ccf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:38.520021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.519909 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:38.997284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.997248 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n"] Oct 30 01:10:38.997948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.997928 4517 cpu_manager.go:414] "containerMap: removing container" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" Oct 30 01:10:38.998059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.997950 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" Oct 30 01:10:38.998059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.997991 4517 cpu_manager.go:414] "containerMap: removing container" podUID="15dc04f8-79b8-41d7-8ada-df7ed4019560" containerName="maas-api" Oct 30 01:10:38.998059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.997998 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15dc04f8-79b8-41d7-8ada-df7ed4019560" containerName="maas-api" Oct 30 01:10:38.998059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.998020 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d34572ac-cb49-4dfd-b400-248d93f99f68" containerName="payload-processing" Oct 30 01:10:38.998059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.998027 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34572ac-cb49-4dfd-b400-248d93f99f68" containerName="payload-processing" Oct 30 01:10:38.998317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.998181 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d4bf7c-8e15-4e8c-9be3-8276d63e9ac8" containerName="istio-proxy" Oct 30 01:10:38.998317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.998202 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="15dc04f8-79b8-41d7-8ada-df7ed4019560" containerName="maas-api" Oct 30 01:10:38.998317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:38.998216 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34572ac-cb49-4dfd-b400-248d93f99f68" containerName="payload-processing" Oct 30 01:10:39.003460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.003431 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.030837 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.030793 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n"] Oct 30 01:10:39.155423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.155383 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.155423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.155430 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.155657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.155560 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3488a16-3617-4db6-ae61-8c93a76f46e1-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.155657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.155607 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.155657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.155646 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.155828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.155803 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrjx\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-kube-api-access-pmrjx\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.155918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.155904 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.155960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.155930 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.155960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.155957 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257214 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257294 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257344 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrjx\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-kube-api-access-pmrjx\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257440 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257474 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257522 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257645 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257682 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257752 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257799 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3488a16-3617-4db6-ae61-8c93a76f46e1-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257906 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.257949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257930 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.258525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.257996 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.258587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.258553 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3488a16-3617-4db6-ae61-8c93a76f46e1-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.260224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.260199 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.260942 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.260918 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.279496 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.279454 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.279839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.279809 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrjx\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-kube-api-access-pmrjx\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.318693 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.318656 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:39.653183 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.653142 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n"] Oct 30 01:10:39.655185 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:10:39.655153 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3488a16_3617_4db6_ae61_8c93a76f46e1.slice/crio-22e9743c8f7af7424d3bdce7f56b81e0599bd43855929135c3832eec3a184261 WatchSource:0}: Error finding container 22e9743c8f7af7424d3bdce7f56b81e0599bd43855929135c3832eec3a184261: Status 404 returned error can't find the container with id 22e9743c8f7af7424d3bdce7f56b81e0599bd43855929135c3832eec3a184261 Oct 30 01:10:39.658623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.658568 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:39.658762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.658683 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:39.658810 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:39.658787 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:10:40.149694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:40.149659 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:40.180264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:40.180224 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" event={"ID":"c3488a16-3617-4db6-ae61-8c93a76f46e1","Type":"ContainerStarted","Data":"8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe"} Oct 30 01:10:40.180472 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:40.180296 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" event={"ID":"c3488a16-3617-4db6-ae61-8c93a76f46e1","Type":"ContainerStarted","Data":"22e9743c8f7af7424d3bdce7f56b81e0599bd43855929135c3832eec3a184261"} Oct 30 01:10:40.239659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:40.239466 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" podStartSLOduration=2.239443859 podStartE2EDuration="2.239443859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:40.238413175 +0000 UTC m=+2178.150540389" watchObservedRunningTime="2026-10-30 01:10:40.239443859 +0000 UTC m=+2178.151571008" Oct 30 01:10:40.319491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:40.319446 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:40.321373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:40.321345 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:10:40.321508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:40.321401 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:10:41.319133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:41.319093 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:10:41.319323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:41.319153 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:10:41.747898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:41.747816 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:42.028968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:42.028886 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr"] Oct 30 01:10:42.039114 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:42.039087 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8ba649-rfkxr"] Oct 30 01:10:42.236753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:42.236697 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43def177-1c54-4500-b25f-faecab3b2af0" path="/var/lib/kubelet/pods/43def177-1c54-4500-b25f-faecab3b2af0/volumes" Oct 30 01:10:42.319619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:42.319580 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:10:42.319809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:42.319631 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:10:43.319434 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:43.319396 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:10:43.319902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:43.319451 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:10:43.723794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:43.723675 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:44.041883 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:44.041846 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r"] Oct 30 01:10:44.044970 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:44.044946 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8ae29d-wpz8r"] Oct 30 01:10:44.236623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:44.236591 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3" path="/var/lib/kubelet/pods/331ad4b0-f70f-4cc6-ad0a-45d3b061a5e3/volumes" Oct 30 01:10:44.319301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:44.319217 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:10:44.319301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:44.319268 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:10:45.319095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:45.319053 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:10:45.319611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:45.319126 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:10:45.500902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:45.500859 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:46.319571 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:46.319520 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:10:46.320054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:46.319587 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:10:47.041772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:47.041695 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:47.325376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:47.325281 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:47.325877 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:47.325594 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:47.326540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:47.326521 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:10:48.353570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:48.353511 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" podUID="25469977-6f4f-4788-ad6a-1f6f91c84ccf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:48.497405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:48.497361 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:49.098196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:49.092242 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:49.364303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:49.364211 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:49.622899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:49.622804 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:49.966487 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:49.966383 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:51.513920 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:51.513851 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:51.677496 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:51.677381 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:51.748986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:51.748958 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:10:51.874896 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:51.874853 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25469977-6f4f-4788-ad6a-1f6f91c84ccf-kube-api-access-49xjw\" (UniqueName: \"kubernetes.io/projected/25469977-6f4f-4788-ad6a-1f6f91c84ccf-kube-api-access-49xjw\") pod \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\" (UID: \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\") " Oct 30 01:10:51.875085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:51.875027 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25469977-6f4f-4788-ad6a-1f6f91c84ccf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25469977-6f4f-4788-ad6a-1f6f91c84ccf-plugins-config-volume\") pod \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\" (UID: \"25469977-6f4f-4788-ad6a-1f6f91c84ccf\") " Oct 30 01:10:51.875444 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:51.875417 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25469977-6f4f-4788-ad6a-1f6f91c84ccf-plugins-config-volume" pod "25469977-6f4f-4788-ad6a-1f6f91c84ccf" (UID: "25469977-6f4f-4788-ad6a-1f6f91c84ccf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:51.875801 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:51.875771 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25469977-6f4f-4788-ad6a-1f6f91c84ccf-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:51.877443 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:51.877413 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25469977-6f4f-4788-ad6a-1f6f91c84ccf-kube-api-access-49xjw" pod "25469977-6f4f-4788-ad6a-1f6f91c84ccf" (UID: "25469977-6f4f-4788-ad6a-1f6f91c84ccf"). InnerVolumeSpecName "kube-api-access-49xjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:51.977362 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:51.977268 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49xjw\" (UniqueName: \"kubernetes.io/projected/25469977-6f4f-4788-ad6a-1f6f91c84ccf-kube-api-access-49xjw\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:10:52.293142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:52.292840 4517 generic.go:356] "Generic (PLEG): container finished" podID="25469977-6f4f-4788-ad6a-1f6f91c84ccf" containerID="d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582" exitCode=0 Oct 30 01:10:52.293142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:52.292890 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" Oct 30 01:10:52.293142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:52.292932 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" event={"ID":"25469977-6f4f-4788-ad6a-1f6f91c84ccf","Type":"ContainerDied","Data":"d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582"} Oct 30 01:10:52.293142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:52.293000 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r" event={"ID":"25469977-6f4f-4788-ad6a-1f6f91c84ccf","Type":"ContainerDied","Data":"b26806f6b1fa42b8051ef35967c2c240467c439adc4aa7142751dd50bec9d139"} Oct 30 01:10:52.293142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:52.293034 4517 scope.go:122] "RemoveContainer" containerID="d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582" Oct 30 01:10:52.306569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:52.306546 4517 scope.go:122] "RemoveContainer" containerID="d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582" Oct 30 01:10:52.306962 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:10:52.306937 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582\": container with ID starting with d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582 not found: ID does not exist" containerID="d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582" Oct 30 01:10:52.307100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:52.306972 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582"} err="failed to get container status \"d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582\": rpc error: code = NotFound desc = could not find container \"d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582\": container with ID starting with d89068a9cf764d48caf71cbaee01a630d0cc03d6de7fe552b055a96876d58582 not found: ID does not exist" Oct 30 01:10:52.343515 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:52.343475 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r"] Oct 30 01:10:52.355927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:52.355884 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f96ede-9b55fbd79-sph6r"] Oct 30 01:10:53.221414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:53.221366 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:54.167270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:54.167231 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:54.238669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:54.238630 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25469977-6f4f-4788-ad6a-1f6f91c84ccf" path="/var/lib/kubelet/pods/25469977-6f4f-4788-ad6a-1f6f91c84ccf/volumes" Oct 30 01:10:54.280120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:54.280077 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:55.016575 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:55.016537 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:56.456359 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:56.456310 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:56.573498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:56.573450 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:56.816741 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:56.816642 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:56.973584 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:56.973537 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:57.588622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:57.588581 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:57.700438 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:57.700395 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:58.084127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:58.084089 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:58.376546 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:58.373940 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:59.215892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:10:59.215846 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:00.425042 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:00.425008 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:00.836510 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:11:00.836473 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:11:01.282982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:01.282901 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:01.941621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:01.941580 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:02.067138 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:02.067083 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:03.220154 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:03.220101 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:03.733070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:03.733009 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:03.848141 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:03.848104 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:04.948877 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:04.948833 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:06.741675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:06.741629 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:08.514544 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:08.514511 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:10.235490 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:10.235440 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:11.644761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:11.644699 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:11.823354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:11.823314 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:13.644823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:13.644792 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:15.041229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:15.041191 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2"] Oct 30 01:11:15.049833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:15.049794 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d50254-449g2"] Oct 30 01:11:15.107866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:15.107819 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:15.227838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:15.227793 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:16.237232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:16.237200 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9efb88a8-9e61-444f-b996-32948d51920b" path="/var/lib/kubelet/pods/9efb88a8-9e61-444f-b996-32948d51920b/volumes" Oct 30 01:11:16.988682 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:16.988645 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:17.119107 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:17.119057 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:18.028657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:18.028618 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj"] Oct 30 01:11:18.039052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:18.039014 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d383e43-ztllj"] Oct 30 01:11:18.236522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:18.236488 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edcfd09-2253-4132-87ef-d38e08356f36" path="/var/lib/kubelet/pods/5edcfd09-2253-4132-87ef-d38e08356f36/volumes" Oct 30 01:11:18.704447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:18.704394 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:20.353006 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:20.352970 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:22.115634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:22.115588 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:23.973875 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:23.973812 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:11:24.891492 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:24.891382 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:26.714034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:26.713976 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:28.615672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:28.615635 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:30.539238 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:30.539203 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:30.762033 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:11:30.761985 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:11:32.275814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:32.275773 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:33.941197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:33.941156 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:34.059265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:34.059223 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:34.464957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:34.464919 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:35.081104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:35.081029 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:35.548205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:35.548159 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:35.823891 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:35.823793 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:35.942142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:35.942107 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:38.151642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:38.151583 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:38.548102 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:38.548056 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:38.679185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:38.679150 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:40.161128 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:40.161085 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:40.360458 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:40.360425 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:40.472403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:40.472293 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:42.299699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:42.299462 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:43.797008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:43.796974 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:45.123229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:45.123188 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:45.638318 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:45.638278 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:47.242008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:47.241934 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:48.948575 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:48.947155 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:49.916385 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:49.916346 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:50.750262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:50.750229 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:52.323838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:52.323789 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5"] Oct 30 01:11:52.370008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:52.369962 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:52.486462 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:52.486425 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:54.150671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:54.150639 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:55.926607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:55.926555 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:57.601002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:57.600956 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:59.185168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:11:59.185131 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:00.761968 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:12:00.761921 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:12:01.025792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:01.025682 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:02.627908 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:02.627872 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:04.294157 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:04.294116 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:05.331149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:05.330841 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:06.341349 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:06.341312 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:07.390478 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.390433 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" podUID="da946f82-5476-4580-b606-53151cf7fa9d" containerName="main" containerID="cri-o://35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a" gracePeriod=45 Oct 30 01:12:07.762764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.762733 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:12:07.800922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.800869 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da946f82-5476-4580-b606-53151cf7fa9d-tls-certs\" (UniqueName: \"kubernetes.io/secret/da946f82-5476-4580-b606-53151cf7fa9d-tls-certs\") pod \"da946f82-5476-4580-b606-53151cf7fa9d\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " Oct 30 01:12:07.801145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.800967 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-home\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-home\") pod \"da946f82-5476-4580-b606-53151cf7fa9d\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " Oct 30 01:12:07.801145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.801023 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-model-cache\") pod \"da946f82-5476-4580-b606-53151cf7fa9d\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " Oct 30 01:12:07.801145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.801107 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-dshm\") pod \"da946f82-5476-4580-b606-53151cf7fa9d\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " Oct 30 01:12:07.801145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.801133 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da946f82-5476-4580-b606-53151cf7fa9d-kube-api-access-nntmp\" (UniqueName: \"kubernetes.io/projected/da946f82-5476-4580-b606-53151cf7fa9d-kube-api-access-nntmp\") pod \"da946f82-5476-4580-b606-53151cf7fa9d\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " Oct 30 01:12:07.801390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.801244 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-tmp-dir\") pod \"da946f82-5476-4580-b606-53151cf7fa9d\" (UID: \"da946f82-5476-4580-b606-53151cf7fa9d\") " Oct 30 01:12:07.801928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.801786 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-model-cache" pod "da946f82-5476-4580-b606-53151cf7fa9d" (UID: "da946f82-5476-4580-b606-53151cf7fa9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:07.804589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.803413 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-home" pod "da946f82-5476-4580-b606-53151cf7fa9d" (UID: "da946f82-5476-4580-b606-53151cf7fa9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:07.804589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.804015 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-tmp-dir" pod "da946f82-5476-4580-b606-53151cf7fa9d" (UID: "da946f82-5476-4580-b606-53151cf7fa9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:07.812075 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.811852 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da946f82-5476-4580-b606-53151cf7fa9d-tls-certs" pod "da946f82-5476-4580-b606-53151cf7fa9d" (UID: "da946f82-5476-4580-b606-53151cf7fa9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:12:07.812327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.812232 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da946f82-5476-4580-b606-53151cf7fa9d-kube-api-access-nntmp" pod "da946f82-5476-4580-b606-53151cf7fa9d" (UID: "da946f82-5476-4580-b606-53151cf7fa9d"). InnerVolumeSpecName "kube-api-access-nntmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:07.812667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.812639 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-dshm" pod "da946f82-5476-4580-b606-53151cf7fa9d" (UID: "da946f82-5476-4580-b606-53151cf7fa9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:07.903264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.903158 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da946f82-5476-4580-b606-53151cf7fa9d-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:12:07.903264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.903205 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:12:07.903264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.903218 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:12:07.903264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.903228 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:12:07.903264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.903240 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nntmp\" (UniqueName: \"kubernetes.io/projected/da946f82-5476-4580-b606-53151cf7fa9d-kube-api-access-nntmp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:12:07.903264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:07.903258 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da946f82-5476-4580-b606-53151cf7fa9d-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:12:08.030355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.030318 4517 generic.go:356] "Generic (PLEG): container finished" podID="da946f82-5476-4580-b606-53151cf7fa9d" containerID="35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a" exitCode=0 Oct 30 01:12:08.030355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.030360 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" Oct 30 01:12:08.030628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.030397 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" event={"ID":"da946f82-5476-4580-b606-53151cf7fa9d","Type":"ContainerDied","Data":"35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a"} Oct 30 01:12:08.030628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.030453 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5" event={"ID":"da946f82-5476-4580-b606-53151cf7fa9d","Type":"ContainerDied","Data":"301af0b1c97208526653738f010b4b76d7a974d8231632227a242ee638cc7c0f"} Oct 30 01:12:08.030628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.030470 4517 scope.go:122] "RemoveContainer" containerID="35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a" Oct 30 01:12:08.042798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.042773 4517 scope.go:122] "RemoveContainer" containerID="35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a" Oct 30 01:12:08.043356 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:12:08.043331 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a\": container with ID starting with 35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a not found: ID does not exist" containerID="35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a" Oct 30 01:12:08.043428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.043367 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a"} err="failed to get container status \"35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a\": rpc error: code = NotFound desc = could not find container \"35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a\": container with ID starting with 35f05de8fbafe798b45d3d40fe9b071d6cf60a47f8d5e8a71bc1c32d7ca2347a not found: ID does not exist" Oct 30 01:12:08.078218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.078170 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5"] Oct 30 01:12:08.089047 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.089016 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/auth-test-model-95492e-kserve-84fd8d947c-tqnq5"] Oct 30 01:12:08.215095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.214963 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:08.239049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:08.239007 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da946f82-5476-4580-b606-53151cf7fa9d" path="/var/lib/kubelet/pods/da946f82-5476-4580-b606-53151cf7fa9d/volumes" Oct 30 01:12:11.393764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:11.393649 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:13.211630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:13.211580 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:15.017331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:15.017292 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:15.152108 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:15.152050 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:15.269554 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:15.269468 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:17.071770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:17.071699 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:17.195985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:17.195919 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:18.450664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:18.450629 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:18.899244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:18.899197 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:19.034333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:19.034295 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:19.363464 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:19.363430 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:19.552295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:19.552253 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:20.675060 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:20.675009 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:22.453005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:22.452963 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:23.175270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:23.175233 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:23.306098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:23.306058 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:24.343368 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:24.343327 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:24.661062 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:24.660971 4517 scope.go:122] "RemoveContainer" containerID="1bf830fd82f4e3f7c02ffc19d8c106be1f75e3b77ae59b691ac3d7c13df68c4c" Oct 30 01:12:24.673371 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:24.673338 4517 scope.go:122] "RemoveContainer" containerID="3fcfb04f12cf3162f0b541f622ad6e13ae1d38f1bb1a9c521bb52e338a29bcff" Oct 30 01:12:24.684388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:24.684350 4517 scope.go:122] "RemoveContainer" containerID="0a7ec28a4c7023868efedafd438d444504d7585404ea7f1b097893ccca887484" Oct 30 01:12:24.696879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:24.696851 4517 scope.go:122] "RemoveContainer" containerID="f13ae32064723e716f7015020b13dd3b12e6b39e41bcf6c63f131b6cab3923ab" Oct 30 01:12:24.918534 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:24.918414 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:25.037331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.037300 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:25.084355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.084313 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf"] Oct 30 01:12:25.084989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.084970 4517 cpu_manager.go:414] "containerMap: removing container" podUID="da946f82-5476-4580-b606-53151cf7fa9d" containerName="main" Oct 30 01:12:25.085076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.084991 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da946f82-5476-4580-b606-53151cf7fa9d" containerName="main" Oct 30 01:12:25.085076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.085023 4517 cpu_manager.go:414] "containerMap: removing container" podUID="25469977-6f4f-4788-ad6a-1f6f91c84ccf" containerName="payload-pre-processing" Oct 30 01:12:25.085076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.085029 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25469977-6f4f-4788-ad6a-1f6f91c84ccf" containerName="payload-pre-processing" Oct 30 01:12:25.085179 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.085124 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="da946f82-5476-4580-b606-53151cf7fa9d" containerName="main" Oct 30 01:12:25.085179 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.085137 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="25469977-6f4f-4788-ad6a-1f6f91c84ccf" containerName="payload-pre-processing" Oct 30 01:12:25.090368 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.090339 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:25.105095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.105068 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f69104\"" Oct 30 01:12:25.105620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.105599 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f69104-dockercfg-2k9lb\"" Oct 30 01:12:25.120090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.120052 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf"] Oct 30 01:12:25.241993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.241857 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844c10a2-f543-4898-b69f-b207052e0d8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf\" (UID: \"844c10a2-f543-4898-b69f-b207052e0d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:25.243525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.243489 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8hs\" (UniqueName: \"kubernetes.io/projected/844c10a2-f543-4898-b69f-b207052e0d8d-kube-api-access-5z8hs\") pod \"payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf\" (UID: \"844c10a2-f543-4898-b69f-b207052e0d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:25.291800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.281833 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj"] Oct 30 01:12:25.314887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.314854 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:25.348727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.348667 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844c10a2-f543-4898-b69f-b207052e0d8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf\" (UID: \"844c10a2-f543-4898-b69f-b207052e0d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:25.349560 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.349527 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8hs\" (UniqueName: \"kubernetes.io/projected/844c10a2-f543-4898-b69f-b207052e0d8d-kube-api-access-5z8hs\") pod \"payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf\" (UID: \"844c10a2-f543-4898-b69f-b207052e0d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:25.354003 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.353968 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844c10a2-f543-4898-b69f-b207052e0d8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf\" (UID: \"844c10a2-f543-4898-b69f-b207052e0d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:25.358631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.358598 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj"] Oct 30 01:12:25.383623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.383545 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8hs\" (UniqueName: \"kubernetes.io/projected/844c10a2-f543-4898-b69f-b207052e0d8d-kube-api-access-5z8hs\") pod \"payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf\" (UID: \"844c10a2-f543-4898-b69f-b207052e0d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:25.408146 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.407740 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:25.453388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.452826 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/3c7ea040-73a1-4179-8b73-4ad5109192a8-kube-api-access-rz2rx\") pod \"payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj\" (UID: \"3c7ea040-73a1-4179-8b73-4ad5109192a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:25.453388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.452974 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7ea040-73a1-4179-8b73-4ad5109192a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj\" (UID: \"3c7ea040-73a1-4179-8b73-4ad5109192a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:25.554290 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.554240 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7ea040-73a1-4179-8b73-4ad5109192a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj\" (UID: \"3c7ea040-73a1-4179-8b73-4ad5109192a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:25.555124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.554565 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/3c7ea040-73a1-4179-8b73-4ad5109192a8-kube-api-access-rz2rx\") pod \"payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj\" (UID: \"3c7ea040-73a1-4179-8b73-4ad5109192a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:25.555425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.555402 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7ea040-73a1-4179-8b73-4ad5109192a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj\" (UID: \"3c7ea040-73a1-4179-8b73-4ad5109192a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:25.582373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.582300 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/3c7ea040-73a1-4179-8b73-4ad5109192a8-kube-api-access-rz2rx\") pod \"payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj\" (UID: \"3c7ea040-73a1-4179-8b73-4ad5109192a8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:25.727909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.727873 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:25.800155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:25.800117 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf"] Oct 30 01:12:26.040571 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.040540 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj"] Oct 30 01:12:26.042750 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:12:26.042679 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7ea040_73a1_4179_8b73_4ad5109192a8.slice/crio-c75c0cd2006d969a5ffde2f4187d5d4b6ee31deda0f5b76a1100a980339d580c WatchSource:0}: Error finding container c75c0cd2006d969a5ffde2f4187d5d4b6ee31deda0f5b76a1100a980339d580c: Status 404 returned error can't find the container with id c75c0cd2006d969a5ffde2f4187d5d4b6ee31deda0f5b76a1100a980339d580c Oct 30 01:12:26.322840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.322744 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" event={"ID":"844c10a2-f543-4898-b69f-b207052e0d8d","Type":"ContainerStarted","Data":"31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50"} Oct 30 01:12:26.322840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.322807 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" event={"ID":"844c10a2-f543-4898-b69f-b207052e0d8d","Type":"ContainerStarted","Data":"f0f41a8b94d2f8e5af70ad37fac17ddfcc52cbf0c39e79702592584619182090"} Oct 30 01:12:26.323113 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.322888 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:26.324954 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.324924 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" event={"ID":"3c7ea040-73a1-4179-8b73-4ad5109192a8","Type":"ContainerStarted","Data":"7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5"} Oct 30 01:12:26.325115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.324987 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:26.325115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.325010 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" event={"ID":"3c7ea040-73a1-4179-8b73-4ad5109192a8","Type":"ContainerStarted","Data":"c75c0cd2006d969a5ffde2f4187d5d4b6ee31deda0f5b76a1100a980339d580c"} Oct 30 01:12:26.331192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.331159 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:26.363631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.363550 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" podStartSLOduration=1.363527964 podStartE2EDuration="1.363527964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:26.362004909 +0000 UTC m=+2284.274132053" watchObservedRunningTime="2026-10-30 01:12:26.363527964 +0000 UTC m=+2284.275655106" Oct 30 01:12:26.413116 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:26.409965 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" podStartSLOduration=1.409940282 podStartE2EDuration="1.409940282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:26.408143237 +0000 UTC m=+2284.320270391" watchObservedRunningTime="2026-10-30 01:12:26.409940282 +0000 UTC m=+2284.322067424" Oct 30 01:12:27.680724 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.680660 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs"] Oct 30 01:12:27.685436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.685401 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.705689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.705628 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-33e759\"/\"kube-root-ca.crt\"" Oct 30 01:12:27.714633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.714598 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-33e759\"/\"openshift-service-ca.crt\"" Oct 30 01:12:27.714633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.714598 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-33e759\"/\"default-dockercfg-6qxp9\"" Oct 30 01:12:27.714925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.714861 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-33e759\"/\"auth-test-model-33e759-kserve-self-signed-certs\"" Oct 30 01:12:27.720833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.720791 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs"] Oct 30 01:12:27.744981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.744947 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:27.789353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.789303 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-tmp-dir\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.789526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.789503 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-model-cache\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.789571 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.789551 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgspj\" (UniqueName: \"kubernetes.io/projected/078c1897-20b9-49d7-8096-d60f593e419e-kube-api-access-sgspj\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.789685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.789662 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-home\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.789781 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.789754 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078c1897-20b9-49d7-8096-d60f593e419e-tls-certs\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.789816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.789799 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-dshm\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.892012 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.891962 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-model-cache\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.892261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.892041 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgspj\" (UniqueName: \"kubernetes.io/projected/078c1897-20b9-49d7-8096-d60f593e419e-kube-api-access-sgspj\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.892261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.892093 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-home\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.892261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.892131 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078c1897-20b9-49d7-8096-d60f593e419e-tls-certs\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.892261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.892168 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-dshm\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.892516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.892378 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-tmp-dir\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.892516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.892496 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-model-cache\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.892635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.892521 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-home\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.892811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.892784 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-tmp-dir\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.894950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.894924 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-dshm\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.895246 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.895226 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078c1897-20b9-49d7-8096-d60f593e419e-tls-certs\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:27.916234 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:27.916192 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgspj\" (UniqueName: \"kubernetes.io/projected/078c1897-20b9-49d7-8096-d60f593e419e-kube-api-access-sgspj\") pod \"auth-test-model-33e759-kserve-64ff8f547c-nd2hs\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:28.008356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.007017 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:28.040735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.030829 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:28.340780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.338672 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b"] Oct 30 01:12:28.344119 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.344090 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.371822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.371441 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b"] Oct 30 01:12:28.432782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.429732 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f823884-3933-465d-a308-c27be99ef1a3-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.432782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.429854 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.432782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.430025 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.432782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.430092 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.432782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.430122 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.432782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.430166 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f823884-3933-465d-a308-c27be99ef1a3-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.432782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.430275 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.432782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.430302 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.432782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.430335 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvw6\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-kube-api-access-xxvw6\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.447001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.445696 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-320045-openshift-default-dockercfg-cbfcp\"" Oct 30 01:12:28.469258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.469229 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs"] Oct 30 01:12:28.471888 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:12:28.471847 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod078c1897_20b9_49d7_8096_d60f593e419e.slice/crio-1d5a0d2f041faceb59c81cca87820439ab8cd0d7ff9e1251c9651cda66524570 WatchSource:0}: Error finding container 1d5a0d2f041faceb59c81cca87820439ab8cd0d7ff9e1251c9651cda66524570: Status 404 returned error can't find the container with id 1d5a0d2f041faceb59c81cca87820439ab8cd0d7ff9e1251c9651cda66524570 Oct 30 01:12:28.531647 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.531602 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.531830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.531689 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.531830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.531744 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.531830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.531798 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f823884-3933-465d-a308-c27be99ef1a3-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.532021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.531897 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.532021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.531928 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.532021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.531958 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvw6\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-kube-api-access-xxvw6\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.532021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.532011 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f823884-3933-465d-a308-c27be99ef1a3-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.532231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.532090 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.532231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.532132 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.532231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.532160 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.532387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.532238 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.532447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.532425 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.533019 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.532991 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f823884-3933-465d-a308-c27be99ef1a3-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.534282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.534254 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.534636 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.534612 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f823884-3933-465d-a308-c27be99ef1a3-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.555606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.555573 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.555879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.555860 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvw6\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-kube-api-access-xxvw6\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:28.678292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:28.678198 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:29.009520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.009487 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b"] Oct 30 01:12:29.011268 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:12:29.011236 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f823884_3933_465d_a308_c27be99ef1a3.slice/crio-cbb844b37a329882230d6065e0b8680f7e4bf6559780c2e026282ba9203e1244 WatchSource:0}: Error finding container cbb844b37a329882230d6065e0b8680f7e4bf6559780c2e026282ba9203e1244: Status 404 returned error can't find the container with id cbb844b37a329882230d6065e0b8680f7e4bf6559780c2e026282ba9203e1244 Oct 30 01:12:29.015361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.015316 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:12:29.015467 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.015403 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:12:29.015467 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.015437 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:12:29.362743 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.362672 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" event={"ID":"078c1897-20b9-49d7-8096-d60f593e419e","Type":"ContainerStarted","Data":"59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b"} Oct 30 01:12:29.363065 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.363029 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" event={"ID":"078c1897-20b9-49d7-8096-d60f593e419e","Type":"ContainerStarted","Data":"1d5a0d2f041faceb59c81cca87820439ab8cd0d7ff9e1251c9651cda66524570"} Oct 30 01:12:29.366615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.366578 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" event={"ID":"3f823884-3933-465d-a308-c27be99ef1a3","Type":"ContainerStarted","Data":"d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a"} Oct 30 01:12:29.366805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.366644 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" event={"ID":"3f823884-3933-465d-a308-c27be99ef1a3","Type":"ContainerStarted","Data":"cbb844b37a329882230d6065e0b8680f7e4bf6559780c2e026282ba9203e1244"} Oct 30 01:12:29.406073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.405996 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" podStartSLOduration=2.405972647 podStartE2EDuration="2.405972647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:29.400059799 +0000 UTC m=+2287.312186964" watchObservedRunningTime="2026-10-30 01:12:29.405972647 +0000 UTC m=+2287.318099790" Oct 30 01:12:29.443269 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.443206 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" podStartSLOduration=1.4431905170000001 podStartE2EDuration="1.443190517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:29.440403867 +0000 UTC m=+2287.352531010" watchObservedRunningTime="2026-10-30 01:12:29.443190517 +0000 UTC m=+2287.355317658" Oct 30 01:12:29.680796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.679823 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:29.808543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:29.808480 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:30.008288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:30.008192 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:30.684947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:30.684861 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:12:30.684947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:30.684939 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" podUID="3f823884-3933-465d-a308-c27be99ef1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 01:12:30.790590 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:12:30.790540 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:12:31.033688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:31.033606 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w"] Oct 30 01:12:31.044006 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:31.043965 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cce7fe-kdc9w"] Oct 30 01:12:31.642265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:31.642228 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:31.682735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:31.682153 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:12:31.682735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:31.682219 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" podUID="3f823884-3933-465d-a308-c27be99ef1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 01:12:32.238132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:32.238090 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30359cf3-e8e1-4df3-9f28-870c3fed5203" path="/var/lib/kubelet/pods/30359cf3-e8e1-4df3-9f28-870c3fed5203/volumes" Oct 30 01:12:32.682734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:32.682220 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:12:32.682734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:32.682296 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" podUID="3f823884-3933-465d-a308-c27be99ef1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 01:12:33.404448 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:33.404408 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:33.679799 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:33.679087 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:12:33.679799 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:33.679174 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" podUID="3f823884-3933-465d-a308-c27be99ef1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 01:12:34.680732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:34.680164 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:12:34.680732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:34.680231 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" podUID="3f823884-3933-465d-a308-c27be99ef1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 01:12:35.131160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:35.131122 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:35.685346 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:35.685304 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:35.686142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:35.685753 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:35.686543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:35.686522 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:12:36.892857 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:36.892819 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:37.337609 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:37.337572 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:12:37.338112 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:37.338089 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:12:38.008102 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:38.008061 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:38.022337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:38.022307 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:38.466791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:38.466745 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:12:38.501973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:38.501938 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:40.134889 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:40.134853 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:41.417284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:41.417238 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:42.016426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:42.016392 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:43.574661 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:43.574622 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:45.317923 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:45.317885 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:46.953290 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:46.953248 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:48.721509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:48.721468 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:50.605887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:50.605826 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:52.436000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:52.435865 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:54.055329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:54.055295 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:54.390644 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:54.390546 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:54.525679 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:54.525647 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:55.040058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:55.040023 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:55.866867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:55.866825 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:57.868738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:57.865210 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:58.560666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:58.560611 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:12:58.733120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:58.733054 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:12:58.854796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:58.853815 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:12:58.878865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:58.878812 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:12:58.882678 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:58.880776 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 01:12:59.225447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:12:59.225299 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:00.761312 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:13:00.761272 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:13:00.912695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:00.912635 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:01.093157 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:01.093120 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:01.210034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:01.209992 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:01.869156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:01.869114 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464"] Oct 30 01:13:01.875649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:01.875611 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:01.889182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:01.889153 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:13:01.889518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:01.889448 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:13:01.892846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:01.892811 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464"] Oct 30 01:13:02.062895 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.062838 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ede1b0c6-hk464\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.063147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.063059 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ede1b0c6-hk464\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.063147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.063122 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5s2q\" (UniqueName: \"kubernetes.io/projected/234df198-0f8d-4ed8-b6ea-6821e65631a8-kube-api-access-h5s2q\") pod \"maas-api-revoke-sub-ede1b0c6-hk464\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.165249 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.165149 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ede1b0c6-hk464\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.166362 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.166332 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ede1b0c6-hk464\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.167920 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.167894 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5s2q\" (UniqueName: \"kubernetes.io/projected/234df198-0f8d-4ed8-b6ea-6821e65631a8-kube-api-access-h5s2q\") pod \"maas-api-revoke-sub-ede1b0c6-hk464\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.169378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.169352 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ede1b0c6-hk464\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.222728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.222058 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ede1b0c6-hk464\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.275553 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.275476 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5s2q\" (UniqueName: \"kubernetes.io/projected/234df198-0f8d-4ed8-b6ea-6821e65631a8-kube-api-access-h5s2q\") pod \"maas-api-revoke-sub-ede1b0c6-hk464\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.497785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.495031 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:02.812654 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.812613 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:02.814747 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:02.814688 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464"] Oct 30 01:13:02.818182 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:13:02.818140 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234df198_0f8d_4ed8_b6ea_6821e65631a8.slice/crio-b6b93dd89fb950af6a85d35ce64465d04ba87574570cafda01c8e60edfbaceb1 WatchSource:0}: Error finding container b6b93dd89fb950af6a85d35ce64465d04ba87574570cafda01c8e60edfbaceb1: Status 404 returned error can't find the container with id b6b93dd89fb950af6a85d35ce64465d04ba87574570cafda01c8e60edfbaceb1 Oct 30 01:13:03.769771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:03.769651 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" event={"ID":"234df198-0f8d-4ed8-b6ea-6821e65631a8","Type":"ContainerStarted","Data":"9dadb2692b2882f138f67f567644f4f7adcb606a79c00c303b28986290adb7d6"} Oct 30 01:13:03.771157 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:03.769816 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" event={"ID":"234df198-0f8d-4ed8-b6ea-6821e65631a8","Type":"ContainerStarted","Data":"b6b93dd89fb950af6a85d35ce64465d04ba87574570cafda01c8e60edfbaceb1"} Oct 30 01:13:03.808393 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:03.808332 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" podStartSLOduration=2.808314323 podStartE2EDuration="2.808314323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:03.806161985 +0000 UTC m=+2321.718289127" watchObservedRunningTime="2026-10-30 01:13:03.808314323 +0000 UTC m=+2321.720441483" Oct 30 01:13:04.743362 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:04.743326 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:04.798673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:04.796863 4517 generic.go:356] "Generic (PLEG): container finished" podID="234df198-0f8d-4ed8-b6ea-6821e65631a8" containerID="9dadb2692b2882f138f67f567644f4f7adcb606a79c00c303b28986290adb7d6" exitCode=0 Oct 30 01:13:04.798673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:04.797138 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" event={"ID":"234df198-0f8d-4ed8-b6ea-6821e65631a8","Type":"ContainerDied","Data":"9dadb2692b2882f138f67f567644f4f7adcb606a79c00c303b28986290adb7d6"} Oct 30 01:13:05.109212 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.109161 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt"] Oct 30 01:13:05.113936 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.113905 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.133100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.133060 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt"] Oct 30 01:13:05.148338 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.148284 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftlj5\" (UniqueName: \"kubernetes.io/projected/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-kube-api-access-ftlj5\") pod \"maas-api-revoke-sub-59216e08-96gbt\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.148559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.148534 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59216e08-96gbt\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.148647 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.148629 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59216e08-96gbt\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.250417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.250361 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftlj5\" (UniqueName: \"kubernetes.io/projected/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-kube-api-access-ftlj5\") pod \"maas-api-revoke-sub-59216e08-96gbt\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.250646 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.250595 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59216e08-96gbt\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.250755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.250650 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59216e08-96gbt\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.251374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.251345 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59216e08-96gbt\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.253504 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.253475 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59216e08-96gbt\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.273505 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.273470 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftlj5\" (UniqueName: \"kubernetes.io/projected/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-kube-api-access-ftlj5\") pod \"maas-api-revoke-sub-59216e08-96gbt\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.429662 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.429566 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:05.747219 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.747189 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt"] Oct 30 01:13:05.749999 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:13:05.749963 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ae8413_1d69_4bbd_a2b8_ad7875dd147b.slice/crio-5afe7ebb5d54014e33157b65d47a6fa14ef56120cfc3d14704d701d91409b15a WatchSource:0}: Error finding container 5afe7ebb5d54014e33157b65d47a6fa14ef56120cfc3d14704d701d91409b15a: Status 404 returned error can't find the container with id 5afe7ebb5d54014e33157b65d47a6fa14ef56120cfc3d14704d701d91409b15a Oct 30 01:13:05.808808 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.808758 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" event={"ID":"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b","Type":"ContainerStarted","Data":"5afe7ebb5d54014e33157b65d47a6fa14ef56120cfc3d14704d701d91409b15a"} Oct 30 01:13:05.977468 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:05.977437 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:06.064561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.064526 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234df198-0f8d-4ed8-b6ea-6821e65631a8-kube-api-access-h5s2q\" (UniqueName: \"kubernetes.io/projected/234df198-0f8d-4ed8-b6ea-6821e65631a8-kube-api-access-h5s2q\") pod \"234df198-0f8d-4ed8-b6ea-6821e65631a8\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " Oct 30 01:13:06.064793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.064627 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-openshift-ca\") pod \"234df198-0f8d-4ed8-b6ea-6821e65631a8\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " Oct 30 01:13:06.064793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.064771 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-xks-ca\") pod \"234df198-0f8d-4ed8-b6ea-6821e65631a8\" (UID: \"234df198-0f8d-4ed8-b6ea-6821e65631a8\") " Oct 30 01:13:06.065283 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.065249 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-openshift-ca" pod "234df198-0f8d-4ed8-b6ea-6821e65631a8" (UID: "234df198-0f8d-4ed8-b6ea-6821e65631a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:06.065554 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.065536 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:06.067414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.067368 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-xks-ca" pod "234df198-0f8d-4ed8-b6ea-6821e65631a8" (UID: "234df198-0f8d-4ed8-b6ea-6821e65631a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:06.067700 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.067684 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234df198-0f8d-4ed8-b6ea-6821e65631a8-kube-api-access-h5s2q" pod "234df198-0f8d-4ed8-b6ea-6821e65631a8" (UID: "234df198-0f8d-4ed8-b6ea-6821e65631a8"). InnerVolumeSpecName "kube-api-access-h5s2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:06.167760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.167698 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5s2q\" (UniqueName: \"kubernetes.io/projected/234df198-0f8d-4ed8-b6ea-6821e65631a8-kube-api-access-h5s2q\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:06.167760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.167758 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234df198-0f8d-4ed8-b6ea-6821e65631a8-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:06.423593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.423489 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:06.820998 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.820965 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" Oct 30 01:13:06.820998 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.820979 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464" event={"ID":"234df198-0f8d-4ed8-b6ea-6821e65631a8","Type":"ContainerDied","Data":"b6b93dd89fb950af6a85d35ce64465d04ba87574570cafda01c8e60edfbaceb1"} Oct 30 01:13:06.821814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.821041 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6b93dd89fb950af6a85d35ce64465d04ba87574570cafda01c8e60edfbaceb1" Oct 30 01:13:06.823223 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.823189 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" event={"ID":"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b","Type":"ContainerStarted","Data":"9b64fe4c73cdfeb5dfde29a6bce533cbe653428d8e0ea9a2ba12dda6e2836f49"} Oct 30 01:13:06.871244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:06.871182 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" podStartSLOduration=1.871166398 podStartE2EDuration="1.871166398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:06.869773744 +0000 UTC m=+2324.781900884" watchObservedRunningTime="2026-10-30 01:13:06.871166398 +0000 UTC m=+2324.783293533" Oct 30 01:13:07.833297 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:07.833258 4517 generic.go:356] "Generic (PLEG): container finished" podID="c7ae8413-1d69-4bbd-a2b8-ad7875dd147b" containerID="9b64fe4c73cdfeb5dfde29a6bce533cbe653428d8e0ea9a2ba12dda6e2836f49" exitCode=0 Oct 30 01:13:07.833757 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:07.833341 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" event={"ID":"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b","Type":"ContainerDied","Data":"9b64fe4c73cdfeb5dfde29a6bce533cbe653428d8e0ea9a2ba12dda6e2836f49"} Oct 30 01:13:08.104008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:08.103916 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:09.026400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.026372 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:09.122973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.122864 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-xks-ca\") pod \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " Oct 30 01:13:09.122973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.122946 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-openshift-ca\") pod \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " Oct 30 01:13:09.123244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.123117 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-kube-api-access-ftlj5\" (UniqueName: \"kubernetes.io/projected/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-kube-api-access-ftlj5\") pod \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\" (UID: \"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b\") " Oct 30 01:13:09.123413 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.123390 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-openshift-ca" pod "c7ae8413-1d69-4bbd-a2b8-ad7875dd147b" (UID: "c7ae8413-1d69-4bbd-a2b8-ad7875dd147b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:09.124038 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.124004 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:09.125521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.125495 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-kube-api-access-ftlj5" pod "c7ae8413-1d69-4bbd-a2b8-ad7875dd147b" (UID: "c7ae8413-1d69-4bbd-a2b8-ad7875dd147b"). InnerVolumeSpecName "kube-api-access-ftlj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:09.125632 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.125546 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-xks-ca" pod "c7ae8413-1d69-4bbd-a2b8-ad7875dd147b" (UID: "c7ae8413-1d69-4bbd-a2b8-ad7875dd147b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:09.225938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.225890 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:09.225938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.225936 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftlj5\" (UniqueName: \"kubernetes.io/projected/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b-kube-api-access-ftlj5\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:09.843775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.843698 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:09.849390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.849365 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" Oct 30 01:13:09.849562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.849407 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt" event={"ID":"c7ae8413-1d69-4bbd-a2b8-ad7875dd147b","Type":"ContainerDied","Data":"5afe7ebb5d54014e33157b65d47a6fa14ef56120cfc3d14704d701d91409b15a"} Oct 30 01:13:09.849562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:09.849484 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5afe7ebb5d54014e33157b65d47a6fa14ef56120cfc3d14704d701d91409b15a" Oct 30 01:13:11.416119 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:11.416084 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:12.267529 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.267497 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn"] Oct 30 01:13:12.268176 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.268159 4517 cpu_manager.go:414] "containerMap: removing container" podUID="234df198-0f8d-4ed8-b6ea-6821e65631a8" containerName="revoke-keys" Oct 30 01:13:12.268253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.268177 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234df198-0f8d-4ed8-b6ea-6821e65631a8" containerName="revoke-keys" Oct 30 01:13:12.268253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.268192 4517 cpu_manager.go:414] "containerMap: removing container" podUID="c7ae8413-1d69-4bbd-a2b8-ad7875dd147b" containerName="revoke-keys" Oct 30 01:13:12.268253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.268197 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ae8413-1d69-4bbd-a2b8-ad7875dd147b" containerName="revoke-keys" Oct 30 01:13:12.268365 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.268297 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="234df198-0f8d-4ed8-b6ea-6821e65631a8" containerName="revoke-keys" Oct 30 01:13:12.268365 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.268306 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ae8413-1d69-4bbd-a2b8-ad7875dd147b" containerName="revoke-keys" Oct 30 01:13:12.272123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.272096 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.286193 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.286163 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:13:12.286422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.286248 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:13:12.297746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.297660 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn"] Oct 30 01:13:12.382202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.382159 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b638c49-dnnhn\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.382427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.382219 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p4b2\" (UniqueName: \"kubernetes.io/projected/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-kube-api-access-2p4b2\") pod \"maas-api-revoke-sub-2b638c49-dnnhn\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.382427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.382270 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b638c49-dnnhn\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.484688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.484647 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b638c49-dnnhn\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.485256 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.484725 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p4b2\" (UniqueName: \"kubernetes.io/projected/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-kube-api-access-2p4b2\") pod \"maas-api-revoke-sub-2b638c49-dnnhn\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.485256 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.484870 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b638c49-dnnhn\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.485579 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.485554 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b638c49-dnnhn\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.487604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.487577 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b638c49-dnnhn\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.509316 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.509281 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p4b2\" (UniqueName: \"kubernetes.io/projected/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-kube-api-access-2p4b2\") pod \"maas-api-revoke-sub-2b638c49-dnnhn\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.612143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.612103 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:12.953530 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:12.953496 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn"] Oct 30 01:13:12.956125 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:13:12.956095 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d16d2e9_91f8_4eca_bd98_2ad3dd604c3c.slice/crio-aa8218132b63af5db85536bed3be07e49d96120899d2ad3f5621668e3b86e7e2 WatchSource:0}: Error finding container aa8218132b63af5db85536bed3be07e49d96120899d2ad3f5621668e3b86e7e2: Status 404 returned error can't find the container with id aa8218132b63af5db85536bed3be07e49d96120899d2ad3f5621668e3b86e7e2 Oct 30 01:13:13.317993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.317964 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:13.471328 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.471274 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr"] Oct 30 01:13:13.476086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.476042 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.502725 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.502668 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr"] Oct 30 01:13:13.608030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.607983 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.608251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.608066 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d793a-45dc-4682-b068-7666d06bb566-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.608251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.608166 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.608251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.608235 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d793a-45dc-4682-b068-7666d06bb566-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.608428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.608384 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.608492 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.608480 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.608551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.608538 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.608610 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.608597 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfvz6\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-kube-api-access-bfvz6\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.608791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.608774 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.709880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.709842 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.709880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.709893 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.709943 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d793a-45dc-4682-b068-7666d06bb566-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.709981 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710041 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d793a-45dc-4682-b068-7666d06bb566-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710131 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710192 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710233 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710265 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfvz6\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-kube-api-access-bfvz6\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710343 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710527 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710608 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710861 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.710966 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.710902 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d793a-45dc-4682-b068-7666d06bb566-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.712545 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.712521 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.712798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.712780 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d793a-45dc-4682-b068-7666d06bb566-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.731839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.731803 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.731996 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.731925 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfvz6\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-kube-api-access-bfvz6\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.818347 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.818298 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:13.885473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.884198 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" event={"ID":"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c","Type":"ContainerStarted","Data":"dd28435aa7c5c445eed4608fef943e180d331cf88688940ca9122ada7d7d13e4"} Oct 30 01:13:13.885473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.884265 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" event={"ID":"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c","Type":"ContainerStarted","Data":"aa8218132b63af5db85536bed3be07e49d96120899d2ad3f5621668e3b86e7e2"} Oct 30 01:13:13.930021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:13.929957 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" podStartSLOduration=1.929939773 podStartE2EDuration="1.929939773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:13.926149609 +0000 UTC m=+2331.838276772" watchObservedRunningTime="2026-10-30 01:13:13.929939773 +0000 UTC m=+2331.842066914" Oct 30 01:13:14.151930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:14.151902 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr"] Oct 30 01:13:14.154408 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:13:14.154375 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39d793a_45dc_4682_b068_7666d06bb566.slice/crio-51582df759613a6084077759f01b2d618fe7260e8a7ae857a174450fed82caf6 WatchSource:0}: Error finding container 51582df759613a6084077759f01b2d618fe7260e8a7ae857a174450fed82caf6: Status 404 returned error can't find the container with id 51582df759613a6084077759f01b2d618fe7260e8a7ae857a174450fed82caf6 Oct 30 01:13:14.157778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:14.157686 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:13:14.157898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:14.157828 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:13:14.157898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:14.157864 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:13:14.895417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:14.895377 4517 generic.go:356] "Generic (PLEG): container finished" podID="4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c" containerID="dd28435aa7c5c445eed4608fef943e180d331cf88688940ca9122ada7d7d13e4" exitCode=0 Oct 30 01:13:14.895954 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:14.895463 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" event={"ID":"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c","Type":"ContainerDied","Data":"dd28435aa7c5c445eed4608fef943e180d331cf88688940ca9122ada7d7d13e4"} Oct 30 01:13:14.897456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:14.897428 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" event={"ID":"b39d793a-45dc-4682-b068-7666d06bb566","Type":"ContainerStarted","Data":"a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438"} Oct 30 01:13:14.897562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:14.897480 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" event={"ID":"b39d793a-45dc-4682-b068-7666d06bb566","Type":"ContainerStarted","Data":"51582df759613a6084077759f01b2d618fe7260e8a7ae857a174450fed82caf6"} Oct 30 01:13:15.040010 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.039968 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:15.496364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.496216 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" podStartSLOduration=2.496189404 podStartE2EDuration="2.496189404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:14.988158387 +0000 UTC m=+2332.900285529" watchObservedRunningTime="2026-10-30 01:13:15.496189404 +0000 UTC m=+2333.408316577" Oct 30 01:13:15.499388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.499347 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj"] Oct 30 01:13:15.504647 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.504617 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.541292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.541250 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj"] Oct 30 01:13:15.645980 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.645930 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q5j2\" (UniqueName: \"kubernetes.io/projected/4dee994c-fb6e-4d1b-bd07-314f8df320d2-kube-api-access-6q5j2\") pod \"maas-api-revoke-sub-f8b4af30-bqzmj\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.646206 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.645998 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b4af30-bqzmj\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.646206 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.646097 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b4af30-bqzmj\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.747378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.747279 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q5j2\" (UniqueName: \"kubernetes.io/projected/4dee994c-fb6e-4d1b-bd07-314f8df320d2-kube-api-access-6q5j2\") pod \"maas-api-revoke-sub-f8b4af30-bqzmj\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.747378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.747329 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b4af30-bqzmj\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.747378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.747356 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b4af30-bqzmj\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.748015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.747984 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b4af30-bqzmj\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.750004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.749983 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b4af30-bqzmj\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.783802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.783632 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q5j2\" (UniqueName: \"kubernetes.io/projected/4dee994c-fb6e-4d1b-bd07-314f8df320d2-kube-api-access-6q5j2\") pod \"maas-api-revoke-sub-f8b4af30-bqzmj\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:15.818528 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.818491 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:15.820468 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.820441 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:13:15.820552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.820488 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 01:13:15.832024 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:15.831991 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:16.206069 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.205921 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:16.207111 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:13:16.207088 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dee994c_fb6e_4d1b_bd07_314f8df320d2.slice/crio-e36e888070a43542790020c149343410d027428c4b5c0996cde3493ca04e059b WatchSource:0}: Error finding container e36e888070a43542790020c149343410d027428c4b5c0996cde3493ca04e059b: Status 404 returned error can't find the container with id e36e888070a43542790020c149343410d027428c4b5c0996cde3493ca04e059b Oct 30 01:13:16.207898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.207874 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj"] Oct 30 01:13:16.255306 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.255273 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-kube-api-access-2p4b2\" (UniqueName: \"kubernetes.io/projected/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-kube-api-access-2p4b2\") pod \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " Oct 30 01:13:16.255508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.255494 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-xks-ca\") pod \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " Oct 30 01:13:16.255602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.255591 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-openshift-ca\") pod \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\" (UID: \"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c\") " Oct 30 01:13:16.256101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.256067 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-openshift-ca" pod "4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c" (UID: "4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:16.256675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.256655 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:16.294059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.285693 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-kube-api-access-2p4b2" pod "4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c" (UID: "4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c"). InnerVolumeSpecName "kube-api-access-2p4b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:16.296508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.296465 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-xks-ca" pod "4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c" (UID: "4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:16.382736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.369919 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p4b2\" (UniqueName: \"kubernetes.io/projected/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-kube-api-access-2p4b2\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:16.382736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.369960 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:16.751958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.751925 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:16.847355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.846854 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:13:16.847355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.846919 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 01:13:16.908276 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.908234 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm"] Oct 30 01:13:16.910573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.910534 4517 cpu_manager.go:414] "containerMap: removing container" podUID="4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c" containerName="revoke-keys" Oct 30 01:13:16.910573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.910572 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c" containerName="revoke-keys" Oct 30 01:13:16.910963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.910822 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c" containerName="revoke-keys" Oct 30 01:13:16.915111 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.915083 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:16.918548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.918515 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" event={"ID":"4dee994c-fb6e-4d1b-bd07-314f8df320d2","Type":"ContainerStarted","Data":"3b106acf8d288431595661551e1af87894e3b5ee8aabda61b5ce157354c4bd22"} Oct 30 01:13:16.918788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.918769 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" event={"ID":"4dee994c-fb6e-4d1b-bd07-314f8df320d2","Type":"ContainerStarted","Data":"e36e888070a43542790020c149343410d027428c4b5c0996cde3493ca04e059b"} Oct 30 01:13:16.920625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.920599 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" Oct 30 01:13:16.920776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.920624 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn" event={"ID":"4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c","Type":"ContainerDied","Data":"aa8218132b63af5db85536bed3be07e49d96120899d2ad3f5621668e3b86e7e2"} Oct 30 01:13:16.920776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.920662 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8218132b63af5db85536bed3be07e49d96120899d2ad3f5621668e3b86e7e2" Oct 30 01:13:16.931865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.931829 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm"] Oct 30 01:13:16.936628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:16.936593 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-62f5d1-openshift-default-dockercfg-hprqt\"" Oct 30 01:13:17.005860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.005810 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-certs\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.006054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.005884 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-credential-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.006142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.006059 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d85cw\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-kube-api-access-d85cw\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.006142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.006127 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.006250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.006222 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-podinfo\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.006305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.006261 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe6ff0-38aa-47bd-977d-12969d6ead64-istiod-ca-cert\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.006360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.006334 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-token\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.006411 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.006397 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-data\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.006464 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.006436 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-envoy\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.028150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.028076 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" podStartSLOduration=2.028051189 podStartE2EDuration="2.028051189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:17.023174206 +0000 UTC m=+2334.935301348" watchObservedRunningTime="2026-10-30 01:13:17.028051189 +0000 UTC m=+2334.940178330" Oct 30 01:13:17.108093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.107994 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-token\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108076 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-data\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108114 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-envoy\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108230 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-certs\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108268 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-credential-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108365 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d85cw\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-kube-api-access-d85cw\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108429 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108505 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-podinfo\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108547 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe6ff0-38aa-47bd-977d-12969d6ead64-istiod-ca-cert\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108640 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-certs\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.108882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.108856 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-data\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.109029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.109001 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.109214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.109195 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-credential-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.109332 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.109310 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe6ff0-38aa-47bd-977d-12969d6ead64-istiod-ca-cert\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.110881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.110856 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-envoy\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.111147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.111130 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-podinfo\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.133964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.133909 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-token\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.134111 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.134045 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d85cw\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-kube-api-access-d85cw\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.236912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.236876 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:17.297452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.297377 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:17.606481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.606444 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm"] Oct 30 01:13:17.616075 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.616034 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:13:17.616224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.616127 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:13:17.616224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.616174 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:13:17.820342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.820304 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:13:17.820697 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.820671 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 01:13:17.932274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.932235 4517 generic.go:356] "Generic (PLEG): container finished" podID="4dee994c-fb6e-4d1b-bd07-314f8df320d2" containerID="3b106acf8d288431595661551e1af87894e3b5ee8aabda61b5ce157354c4bd22" exitCode=0 Oct 30 01:13:17.932491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.932321 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" event={"ID":"4dee994c-fb6e-4d1b-bd07-314f8df320d2","Type":"ContainerDied","Data":"3b106acf8d288431595661551e1af87894e3b5ee8aabda61b5ce157354c4bd22"} Oct 30 01:13:17.934535 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.934500 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" event={"ID":"04fe6ff0-38aa-47bd-977d-12969d6ead64","Type":"ContainerStarted","Data":"eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8"} Oct 30 01:13:17.934697 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:17.934559 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" event={"ID":"04fe6ff0-38aa-47bd-977d-12969d6ead64","Type":"ContainerStarted","Data":"ec56a28e7618b52b9c1135f128f677d75e02d9406c2d16e63e0e9f0004e83b37"} Oct 30 01:13:18.044027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:18.043971 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" podStartSLOduration=2.043952457 podStartE2EDuration="2.043952457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:18.040763651 +0000 UTC m=+2335.952890791" watchObservedRunningTime="2026-10-30 01:13:18.043952457 +0000 UTC m=+2335.956079599" Oct 30 01:13:18.241271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:18.241234 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:18.245983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:18.245945 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:13:18.246180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:18.246021 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:13:18.582885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:18.582842 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:18.820116 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:18.819316 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:13:18.820116 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:18.819372 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 01:13:19.237568 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.237448 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:13:19.237918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.237860 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:13:19.339445 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.339417 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:19.456160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.456123 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-xks-ca\") pod \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " Oct 30 01:13:19.456379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.456241 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dee994c-fb6e-4d1b-bd07-314f8df320d2-kube-api-access-6q5j2\" (UniqueName: \"kubernetes.io/projected/4dee994c-fb6e-4d1b-bd07-314f8df320d2-kube-api-access-6q5j2\") pod \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " Oct 30 01:13:19.456379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.456267 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-openshift-ca\") pod \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\" (UID: \"4dee994c-fb6e-4d1b-bd07-314f8df320d2\") " Oct 30 01:13:19.456879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.456848 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-openshift-ca" pod "4dee994c-fb6e-4d1b-bd07-314f8df320d2" (UID: "4dee994c-fb6e-4d1b-bd07-314f8df320d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:19.528796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.527887 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-xks-ca" pod "4dee994c-fb6e-4d1b-bd07-314f8df320d2" (UID: "4dee994c-fb6e-4d1b-bd07-314f8df320d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:19.536870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.534905 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dee994c-fb6e-4d1b-bd07-314f8df320d2-kube-api-access-6q5j2" pod "4dee994c-fb6e-4d1b-bd07-314f8df320d2" (UID: "4dee994c-fb6e-4d1b-bd07-314f8df320d2"). InnerVolumeSpecName "kube-api-access-6q5j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:19.581752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.558581 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q5j2\" (UniqueName: \"kubernetes.io/projected/4dee994c-fb6e-4d1b-bd07-314f8df320d2-kube-api-access-6q5j2\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:19.581752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.558621 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:19.581752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.558636 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dee994c-fb6e-4d1b-bd07-314f8df320d2-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:19.819737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.819433 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:13:19.819737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.819509 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 01:13:19.951873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.951830 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" event={"ID":"4dee994c-fb6e-4d1b-bd07-314f8df320d2","Type":"ContainerDied","Data":"e36e888070a43542790020c149343410d027428c4b5c0996cde3493ca04e059b"} Oct 30 01:13:19.952090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.951894 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e36e888070a43542790020c149343410d027428c4b5c0996cde3493ca04e059b" Oct 30 01:13:19.952090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:19.951850 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj" Oct 30 01:13:20.231266 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:20.231165 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:20.237959 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:20.237919 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:13:20.238167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:20.238011 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:13:20.819189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:20.819147 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:13:20.819632 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:20.819216 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 01:13:21.237358 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:21.237257 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:13:21.237358 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:21.237314 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:13:21.824289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:21.824256 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:21.824847 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:21.824616 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:21.825596 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:21.825572 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:13:21.902224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:21.902175 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:22.237496 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:22.237418 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:13:22.237496 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:22.237480 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:13:23.237289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:23.237252 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:13:23.237855 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:23.237820 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:13:23.628094 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:23.628007 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:24.237539 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:24.237503 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:13:24.238020 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:24.237558 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:13:24.752564 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:24.752521 4517 scope.go:122] "RemoveContainer" containerID="918ceb1abde4bdd3c4a291176cad28c10067985c68e6b307765e65c5ca2e8888" Oct 30 01:13:24.765826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:24.765793 4517 scope.go:122] "RemoveContainer" containerID="64eabba66543d989acc35a5d362645f07c414f1208137b1f493d8e13955c14a1" Oct 30 01:13:24.779783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:24.779749 4517 scope.go:122] "RemoveContainer" containerID="64f017377cd1ea9b078b7190a67f11f3d9dd0e9e8ab4a4e28aa03feddb96834f" Oct 30 01:13:24.793464 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:24.793444 4517 scope.go:122] "RemoveContainer" containerID="66e3648bd76c71390b7ee5a01c3224bbca08f32bab3f572e92210426a696fed3" Oct 30 01:13:25.242771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:25.242739 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:25.243278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:25.243263 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:25.244433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:25.244383 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:13:25.340738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:25.340196 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:25.508863 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:25.500375 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:27.042415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:27.042363 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:28.945228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:28.945184 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:30.762497 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:13:30.762459 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:13:31.106290 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:31.106254 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:32.692281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:32.692239 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:32.812397 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:32.812360 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:33.782244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.782204 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8991-740c4c"] Oct 30 01:13:33.783083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.782825 4517 cpu_manager.go:414] "containerMap: removing container" podUID="4dee994c-fb6e-4d1b-bd07-314f8df320d2" containerName="revoke-keys" Oct 30 01:13:33.783083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.782839 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dee994c-fb6e-4d1b-bd07-314f8df320d2" containerName="revoke-keys" Oct 30 01:13:33.783083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.782994 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dee994c-fb6e-4d1b-bd07-314f8df320d2" containerName="revoke-keys" Oct 30 01:13:33.789167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.789128 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-740c4c" Oct 30 01:13:33.805670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.805628 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-740c4c"] Oct 30 01:13:33.810369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.810339 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v4hbx\"" Oct 30 01:13:33.852608 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.852574 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb7hx\" (UniqueName: \"kubernetes.io/projected/3142ce41-edf9-42f0-96bb-b0b19b97b3bf-kube-api-access-tb7hx\") pod \"test-curl-8991-740c4c\" (UID: \"3142ce41-edf9-42f0-96bb-b0b19b97b3bf\") " pod="opendatahub/test-curl-8991-740c4c" Oct 30 01:13:33.954143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.954099 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7hx\" (UniqueName: \"kubernetes.io/projected/3142ce41-edf9-42f0-96bb-b0b19b97b3bf-kube-api-access-tb7hx\") pod \"test-curl-8991-740c4c\" (UID: \"3142ce41-edf9-42f0-96bb-b0b19b97b3bf\") " pod="opendatahub/test-curl-8991-740c4c" Oct 30 01:13:33.981823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:33.981789 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7hx\" (UniqueName: \"kubernetes.io/projected/3142ce41-edf9-42f0-96bb-b0b19b97b3bf-kube-api-access-tb7hx\") pod \"test-curl-8991-740c4c\" (UID: \"3142ce41-edf9-42f0-96bb-b0b19b97b3bf\") " pod="opendatahub/test-curl-8991-740c4c" Oct 30 01:13:34.106668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:34.106611 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-740c4c" Oct 30 01:13:34.420525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:34.420492 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-740c4c"] Oct 30 01:13:34.422452 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:13:34.422411 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3142ce41_edf9_42f0_96bb_b0b19b97b3bf.slice/crio-3055de626a20a2aed3bdb4593b7096e8b8422ca00a6df1b8a4b03c85f71c87e4 WatchSource:0}: Error finding container 3055de626a20a2aed3bdb4593b7096e8b8422ca00a6df1b8a4b03c85f71c87e4: Status 404 returned error can't find the container with id 3055de626a20a2aed3bdb4593b7096e8b8422ca00a6df1b8a4b03c85f71c87e4 Oct 30 01:13:34.593182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:34.593150 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:35.234467 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:35.234429 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-740c4c" event={"ID":"3142ce41-edf9-42f0-96bb-b0b19b97b3bf","Type":"ContainerStarted","Data":"bc8eb918eb66a2ecafe90f53fab34c6ad2db17dc3410cc42f68747f94392f7b6"} Oct 30 01:13:35.234982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:35.234477 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-740c4c" event={"ID":"3142ce41-edf9-42f0-96bb-b0b19b97b3bf","Type":"ContainerStarted","Data":"3055de626a20a2aed3bdb4593b7096e8b8422ca00a6df1b8a4b03c85f71c87e4"} Oct 30 01:13:35.271895 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:35.271831 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8991-740c4c" podStartSLOduration=2.271812113 podStartE2EDuration="2.271812113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:35.269446822 +0000 UTC m=+2353.181573964" watchObservedRunningTime="2026-10-30 01:13:35.271812113 +0000 UTC m=+2353.183939265" Oct 30 01:13:36.218190 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.218155 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8991-740c4c"] Oct 30 01:13:36.224770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.223689 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8991-740c4c"] Oct 30 01:13:36.259456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.259419 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:36.375532 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.375489 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8991-400514"] Oct 30 01:13:36.376508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.376477 4517 cpu_manager.go:414] "containerMap: removing container" podUID="3142ce41-edf9-42f0-96bb-b0b19b97b3bf" containerName="test-curl-8991-740c4c" Oct 30 01:13:36.376508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.376507 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3142ce41-edf9-42f0-96bb-b0b19b97b3bf" containerName="test-curl-8991-740c4c" Oct 30 01:13:36.376794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.376774 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="3142ce41-edf9-42f0-96bb-b0b19b97b3bf" containerName="test-curl-8991-740c4c" Oct 30 01:13:36.380506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.380477 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-400514" Oct 30 01:13:36.396480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.396449 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-400514"] Oct 30 01:13:36.495940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.495846 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzqc8\" (UniqueName: \"kubernetes.io/projected/65684e08-105a-464f-a816-11565a127bc5-kube-api-access-kzqc8\") pod \"test-curl-8991-400514\" (UID: \"65684e08-105a-464f-a816-11565a127bc5\") " pod="opendatahub/test-curl-8991-400514" Oct 30 01:13:36.597403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.597355 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqc8\" (UniqueName: \"kubernetes.io/projected/65684e08-105a-464f-a816-11565a127bc5-kube-api-access-kzqc8\") pod \"test-curl-8991-400514\" (UID: \"65684e08-105a-464f-a816-11565a127bc5\") " pod="opendatahub/test-curl-8991-400514" Oct 30 01:13:36.618688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.618652 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqc8\" (UniqueName: \"kubernetes.io/projected/65684e08-105a-464f-a816-11565a127bc5-kube-api-access-kzqc8\") pod \"test-curl-8991-400514\" (UID: \"65684e08-105a-464f-a816-11565a127bc5\") " pod="opendatahub/test-curl-8991-400514" Oct 30 01:13:36.697431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:36.697390 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-400514" Oct 30 01:13:37.001620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:37.001588 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8991-400514"] Oct 30 01:13:37.003768 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:13:37.003657 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65684e08_105a_464f_a816_11565a127bc5.slice/crio-056133b4f2019b87242aa0ec8a16c454c7653e3be5f812fa4e2fb50f25afbc43 WatchSource:0}: Error finding container 056133b4f2019b87242aa0ec8a16c454c7653e3be5f812fa4e2fb50f25afbc43: Status 404 returned error can't find the container with id 056133b4f2019b87242aa0ec8a16c454c7653e3be5f812fa4e2fb50f25afbc43 Oct 30 01:13:37.255879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:37.255771 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-400514" event={"ID":"65684e08-105a-464f-a816-11565a127bc5","Type":"ContainerStarted","Data":"b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b"} Oct 30 01:13:37.255879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:37.255825 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8991-400514" event={"ID":"65684e08-105a-464f-a816-11565a127bc5","Type":"ContainerStarted","Data":"056133b4f2019b87242aa0ec8a16c454c7653e3be5f812fa4e2fb50f25afbc43"} Oct 30 01:13:37.256124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:37.255936 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8991-740c4c" podUID="3142ce41-edf9-42f0-96bb-b0b19b97b3bf" containerName="test-curl-8991-740c4c" containerID="cri-o://bc8eb918eb66a2ecafe90f53fab34c6ad2db17dc3410cc42f68747f94392f7b6" gracePeriod=2 Oct 30 01:13:37.296301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:37.296225 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8991-400514" podStartSLOduration=1.296203443 podStartE2EDuration="1.296203443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:37.293875498 +0000 UTC m=+2355.206002660" watchObservedRunningTime="2026-10-30 01:13:37.296203443 +0000 UTC m=+2355.208330585" Oct 30 01:13:38.032739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.032682 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:38.183743 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.180810 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8991-400514"] Oct 30 01:13:38.198737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.193472 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8991-400514"] Oct 30 01:13:38.198737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.195819 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:38.717231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.717194 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:38.722787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.722752 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn"] Oct 30 01:13:38.723469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.723447 4517 cpu_manager.go:414] "containerMap: removing container" podUID="65684e08-105a-464f-a816-11565a127bc5" containerName="test-curl-8991-400514" Oct 30 01:13:38.723584 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.723469 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65684e08-105a-464f-a816-11565a127bc5" containerName="test-curl-8991-400514" Oct 30 01:13:38.723760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.723700 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="65684e08-105a-464f-a816-11565a127bc5" containerName="test-curl-8991-400514" Oct 30 01:13:38.727165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.727138 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:38.746582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.746193 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn"] Oct 30 01:13:38.747380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.747338 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c522237d-hmqcn\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:38.747550 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.747505 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q624q\" (UniqueName: \"kubernetes.io/projected/6df32418-cac6-433c-a7d6-285e12a053f4-kube-api-access-q624q\") pod \"maas-api-revoke-sub-c522237d-hmqcn\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:38.747629 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.747575 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c522237d-hmqcn\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:38.751699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.751666 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:13:38.751885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.751819 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:13:38.849562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.849510 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q624q\" (UniqueName: \"kubernetes.io/projected/6df32418-cac6-433c-a7d6-285e12a053f4-kube-api-access-q624q\") pod \"maas-api-revoke-sub-c522237d-hmqcn\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:38.849815 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.849673 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c522237d-hmqcn\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:38.850000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.849980 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c522237d-hmqcn\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:38.850580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.850561 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c522237d-hmqcn\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:38.852563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.852542 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c522237d-hmqcn\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:38.872224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:38.872190 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q624q\" (UniqueName: \"kubernetes.io/projected/6df32418-cac6-433c-a7d6-285e12a053f4-kube-api-access-q624q\") pod \"maas-api-revoke-sub-c522237d-hmqcn\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:39.042130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:39.042090 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:39.200977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:39.178952 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:39.331819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:39.331347 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:39.483739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:39.483033 4517 generic.go:356] "Generic (PLEG): container finished" podID="3142ce41-edf9-42f0-96bb-b0b19b97b3bf" containerID="bc8eb918eb66a2ecafe90f53fab34c6ad2db17dc3410cc42f68747f94392f7b6" exitCode=137 Oct 30 01:13:39.483739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:39.483227 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8991-400514" podUID="65684e08-105a-464f-a816-11565a127bc5" containerName="test-curl-8991-400514" containerID="cri-o://b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b" gracePeriod=2 Oct 30 01:13:39.567295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:39.567254 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn"] Oct 30 01:13:39.629737 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:13:39.626963 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df32418_cac6_433c_a7d6_285e12a053f4.slice/crio-176b792bd6c43ec86b9e840ee6925305dd7784f4a154cdb43c740afd5fe140e8 WatchSource:0}: Error finding container 176b792bd6c43ec86b9e840ee6925305dd7784f4a154cdb43c740afd5fe140e8: Status 404 returned error can't find the container with id 176b792bd6c43ec86b9e840ee6925305dd7784f4a154cdb43c740afd5fe140e8 Oct 30 01:13:39.794410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:39.794375 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-740c4c" Oct 30 01:13:39.908076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:39.908040 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3142ce41-edf9-42f0-96bb-b0b19b97b3bf-kube-api-access-tb7hx\" (UniqueName: \"kubernetes.io/projected/3142ce41-edf9-42f0-96bb-b0b19b97b3bf-kube-api-access-tb7hx\") pod \"3142ce41-edf9-42f0-96bb-b0b19b97b3bf\" (UID: \"3142ce41-edf9-42f0-96bb-b0b19b97b3bf\") " Oct 30 01:13:39.910620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:39.910582 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3142ce41-edf9-42f0-96bb-b0b19b97b3bf-kube-api-access-tb7hx" pod "3142ce41-edf9-42f0-96bb-b0b19b97b3bf" (UID: "3142ce41-edf9-42f0-96bb-b0b19b97b3bf"). InnerVolumeSpecName "kube-api-access-tb7hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:40.010192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:40.010143 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb7hx\" (UniqueName: \"kubernetes.io/projected/3142ce41-edf9-42f0-96bb-b0b19b97b3bf-kube-api-access-tb7hx\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:40.021927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:40.021891 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:40.238869 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:40.238832 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3142ce41-edf9-42f0-96bb-b0b19b97b3bf" path="/var/lib/kubelet/pods/3142ce41-edf9-42f0-96bb-b0b19b97b3bf/volumes" Oct 30 01:13:40.496254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:40.496153 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" event={"ID":"6df32418-cac6-433c-a7d6-285e12a053f4","Type":"ContainerStarted","Data":"ba4e9ac6b3c6ac7d2944fb4a41e84c0906ceac247f707a90e55fc416b98dab70"} Oct 30 01:13:40.496254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:40.496214 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" event={"ID":"6df32418-cac6-433c-a7d6-285e12a053f4","Type":"ContainerStarted","Data":"176b792bd6c43ec86b9e840ee6925305dd7784f4a154cdb43c740afd5fe140e8"} Oct 30 01:13:40.498138 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:40.498109 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-740c4c" Oct 30 01:13:40.498327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:40.498162 4517 scope.go:122] "RemoveContainer" containerID="bc8eb918eb66a2ecafe90f53fab34c6ad2db17dc3410cc42f68747f94392f7b6" Oct 30 01:13:40.539922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:40.539847 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" podStartSLOduration=2.539825134 podStartE2EDuration="2.539825134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:40.53291207 +0000 UTC m=+2358.445039235" watchObservedRunningTime="2026-10-30 01:13:40.539825134 +0000 UTC m=+2358.451952277" Oct 30 01:13:41.509877 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:41.509843 4517 generic.go:356] "Generic (PLEG): container finished" podID="6df32418-cac6-433c-a7d6-285e12a053f4" containerID="ba4e9ac6b3c6ac7d2944fb4a41e84c0906ceac247f707a90e55fc416b98dab70" exitCode=0 Oct 30 01:13:41.510258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:41.509984 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" event={"ID":"6df32418-cac6-433c-a7d6-285e12a053f4","Type":"ContainerDied","Data":"ba4e9ac6b3c6ac7d2944fb4a41e84c0906ceac247f707a90e55fc416b98dab70"} Oct 30 01:13:42.050405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.050371 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-400514" Oct 30 01:13:42.158394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.158352 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65684e08-105a-464f-a816-11565a127bc5-kube-api-access-kzqc8\" (UniqueName: \"kubernetes.io/projected/65684e08-105a-464f-a816-11565a127bc5-kube-api-access-kzqc8\") pod \"65684e08-105a-464f-a816-11565a127bc5\" (UID: \"65684e08-105a-464f-a816-11565a127bc5\") " Oct 30 01:13:42.168853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.162617 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65684e08-105a-464f-a816-11565a127bc5-kube-api-access-kzqc8" pod "65684e08-105a-464f-a816-11565a127bc5" (UID: "65684e08-105a-464f-a816-11565a127bc5"). InnerVolumeSpecName "kube-api-access-kzqc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:42.241062 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.240965 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65684e08-105a-464f-a816-11565a127bc5" path="/var/lib/kubelet/pods/65684e08-105a-464f-a816-11565a127bc5/volumes" Oct 30 01:13:42.260818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.260773 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzqc8\" (UniqueName: \"kubernetes.io/projected/65684e08-105a-464f-a816-11565a127bc5-kube-api-access-kzqc8\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:42.300582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.300538 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:42.519413 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.519289 4517 generic.go:356] "Generic (PLEG): container finished" podID="65684e08-105a-464f-a816-11565a127bc5" containerID="b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b" exitCode=137 Oct 30 01:13:42.519413 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.519335 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8991-400514" Oct 30 01:13:42.519413 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.519397 4517 scope.go:122] "RemoveContainer" containerID="b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b" Oct 30 01:13:42.530547 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.530513 4517 scope.go:122] "RemoveContainer" containerID="b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b" Oct 30 01:13:42.531001 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:13:42.530977 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b\": container with ID starting with b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b not found: ID does not exist" containerID="b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b" Oct 30 01:13:42.531089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.531010 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b"} err="failed to get container status \"b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b\": rpc error: code = NotFound desc = could not find container \"b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b\": container with ID starting with b255af91da569075fc254ed793de53aa141d9873c5956be0e1e8f8b22c5a723b not found: ID does not exist" Oct 30 01:13:42.707993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.707958 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:42.773313 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.773209 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-xks-ca\") pod \"6df32418-cac6-433c-a7d6-285e12a053f4\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " Oct 30 01:13:42.773313 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.773270 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df32418-cac6-433c-a7d6-285e12a053f4-kube-api-access-q624q\" (UniqueName: \"kubernetes.io/projected/6df32418-cac6-433c-a7d6-285e12a053f4-kube-api-access-q624q\") pod \"6df32418-cac6-433c-a7d6-285e12a053f4\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " Oct 30 01:13:42.773578 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.773428 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-openshift-ca\") pod \"6df32418-cac6-433c-a7d6-285e12a053f4\" (UID: \"6df32418-cac6-433c-a7d6-285e12a053f4\") " Oct 30 01:13:42.773911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.773881 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-openshift-ca" pod "6df32418-cac6-433c-a7d6-285e12a053f4" (UID: "6df32418-cac6-433c-a7d6-285e12a053f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:42.775012 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.774982 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:42.775788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.775758 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-xks-ca" pod "6df32418-cac6-433c-a7d6-285e12a053f4" (UID: "6df32418-cac6-433c-a7d6-285e12a053f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:42.775963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.775942 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df32418-cac6-433c-a7d6-285e12a053f4-kube-api-access-q624q" pod "6df32418-cac6-433c-a7d6-285e12a053f4" (UID: "6df32418-cac6-433c-a7d6-285e12a053f4"). InnerVolumeSpecName "kube-api-access-q624q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:42.876540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.876509 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df32418-cac6-433c-a7d6-285e12a053f4-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:42.876540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:42.876538 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q624q\" (UniqueName: \"kubernetes.io/projected/6df32418-cac6-433c-a7d6-285e12a053f4-kube-api-access-q624q\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.527948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:43.527904 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" Oct 30 01:13:43.528480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:43.527944 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn" event={"ID":"6df32418-cac6-433c-a7d6-285e12a053f4","Type":"ContainerDied","Data":"176b792bd6c43ec86b9e840ee6925305dd7784f4a154cdb43c740afd5fe140e8"} Oct 30 01:13:43.528480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:43.528003 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="176b792bd6c43ec86b9e840ee6925305dd7784f4a154cdb43c740afd5fe140e8" Oct 30 01:13:44.239424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:44.239384 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:44.445585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:44.445550 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:44.562366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:44.562320 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:44.879677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:44.879560 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:45.190449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:45.190346 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:46.473076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:46.473022 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:46.866189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:46.866144 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:46.999407 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:46.999356 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:48.183986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:48.183950 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:49.031465 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.031423 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5"] Oct 30 01:13:49.032449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.032427 4517 cpu_manager.go:414] "containerMap: removing container" podUID="6df32418-cac6-433c-a7d6-285e12a053f4" containerName="revoke-keys" Oct 30 01:13:49.032449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.032450 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df32418-cac6-433c-a7d6-285e12a053f4" containerName="revoke-keys" Oct 30 01:13:49.032588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.032562 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df32418-cac6-433c-a7d6-285e12a053f4" containerName="revoke-keys" Oct 30 01:13:49.036486 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.036462 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.061527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.061486 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5"] Oct 30 01:13:49.066936 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.066870 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:13:49.067760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.067734 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:13:49.120450 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.120412 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mk2d\" (UniqueName: \"kubernetes.io/projected/67717cb8-67ca-422d-ac77-77be7d572941-kube-api-access-4mk2d\") pod \"maas-api-revoke-sub-451bb244-z4vc5\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.120654 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.120608 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451bb244-z4vc5\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.120823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.120805 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451bb244-z4vc5\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.221944 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.221906 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451bb244-z4vc5\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.222446 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.222052 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451bb244-z4vc5\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.222446 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.222102 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mk2d\" (UniqueName: \"kubernetes.io/projected/67717cb8-67ca-422d-ac77-77be7d572941-kube-api-access-4mk2d\") pod \"maas-api-revoke-sub-451bb244-z4vc5\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.222798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.222778 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451bb244-z4vc5\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.224627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.224596 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451bb244-z4vc5\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.243699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.243663 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mk2d\" (UniqueName: \"kubernetes.io/projected/67717cb8-67ca-422d-ac77-77be7d572941-kube-api-access-4mk2d\") pod \"maas-api-revoke-sub-451bb244-z4vc5\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.351849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.351815 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:49.675608 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:49.675571 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5"] Oct 30 01:13:49.679132 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:13:49.679089 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67717cb8_67ca_422d_ac77_77be7d572941.slice/crio-1fe225613be4e8843eee0c7492d8e1a1246454acfe4b0e15346468ca9e27bf27 WatchSource:0}: Error finding container 1fe225613be4e8843eee0c7492d8e1a1246454acfe4b0e15346468ca9e27bf27: Status 404 returned error can't find the container with id 1fe225613be4e8843eee0c7492d8e1a1246454acfe4b0e15346468ca9e27bf27 Oct 30 01:13:50.118383 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:50.118342 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:50.603884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:50.603831 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" event={"ID":"67717cb8-67ca-422d-ac77-77be7d572941","Type":"ContainerStarted","Data":"91f433a2cea2af8a0b34c6cec87dbdad6e0787b4150d96344b2a160fcb01b1ba"} Oct 30 01:13:50.603884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:50.603894 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" event={"ID":"67717cb8-67ca-422d-ac77-77be7d572941","Type":"ContainerStarted","Data":"1fe225613be4e8843eee0c7492d8e1a1246454acfe4b0e15346468ca9e27bf27"} Oct 30 01:13:50.642662 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:50.642575 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" podStartSLOduration=1.642553486 podStartE2EDuration="1.642553486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:50.641206447 +0000 UTC m=+2368.553333587" watchObservedRunningTime="2026-10-30 01:13:50.642553486 +0000 UTC m=+2368.554680628" Oct 30 01:13:51.615949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:51.615915 4517 generic.go:356] "Generic (PLEG): container finished" podID="67717cb8-67ca-422d-ac77-77be7d572941" containerID="91f433a2cea2af8a0b34c6cec87dbdad6e0787b4150d96344b2a160fcb01b1ba" exitCode=0 Oct 30 01:13:51.616391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:51.616000 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" event={"ID":"67717cb8-67ca-422d-ac77-77be7d572941","Type":"ContainerDied","Data":"91f433a2cea2af8a0b34c6cec87dbdad6e0787b4150d96344b2a160fcb01b1ba"} Oct 30 01:13:51.776775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:51.776701 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:52.819174 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:52.819142 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:52.983665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:52.983572 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67717cb8-67ca-422d-ac77-77be7d572941-kube-api-access-4mk2d\" (UniqueName: \"kubernetes.io/projected/67717cb8-67ca-422d-ac77-77be7d572941-kube-api-access-4mk2d\") pod \"67717cb8-67ca-422d-ac77-77be7d572941\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " Oct 30 01:13:52.983850 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:52.983809 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-xks-ca\") pod \"67717cb8-67ca-422d-ac77-77be7d572941\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " Oct 30 01:13:52.983850 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:52.983840 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-openshift-ca\") pod \"67717cb8-67ca-422d-ac77-77be7d572941\" (UID: \"67717cb8-67ca-422d-ac77-77be7d572941\") " Oct 30 01:13:52.984283 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:52.984253 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-openshift-ca" pod "67717cb8-67ca-422d-ac77-77be7d572941" (UID: "67717cb8-67ca-422d-ac77-77be7d572941"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:52.984773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:52.984747 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:52.986120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:52.986094 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-xks-ca" pod "67717cb8-67ca-422d-ac77-77be7d572941" (UID: "67717cb8-67ca-422d-ac77-77be7d572941"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:52.986857 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:52.986835 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67717cb8-67ca-422d-ac77-77be7d572941-kube-api-access-4mk2d" pod "67717cb8-67ca-422d-ac77-77be7d572941" (UID: "67717cb8-67ca-422d-ac77-77be7d572941"). InnerVolumeSpecName "kube-api-access-4mk2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:53.086630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:53.086584 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mk2d\" (UniqueName: \"kubernetes.io/projected/67717cb8-67ca-422d-ac77-77be7d572941-kube-api-access-4mk2d\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:53.086630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:53.086632 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67717cb8-67ca-422d-ac77-77be7d572941-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:13:53.498840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:53.498799 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:53.646253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:53.646224 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" Oct 30 01:13:53.646420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:53.646222 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5" event={"ID":"67717cb8-67ca-422d-ac77-77be7d572941","Type":"ContainerDied","Data":"1fe225613be4e8843eee0c7492d8e1a1246454acfe4b0e15346468ca9e27bf27"} Oct 30 01:13:53.646420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:53.646366 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fe225613be4e8843eee0c7492d8e1a1246454acfe4b0e15346468ca9e27bf27" Oct 30 01:13:55.330402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:55.330350 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:56.943758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:56.943670 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:57.465782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:57.465736 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:58.533895 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:58.533852 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:59.853044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.853006 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws"] Oct 30 01:13:59.853917 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.853893 4517 cpu_manager.go:414] "containerMap: removing container" podUID="67717cb8-67ca-422d-ac77-77be7d572941" containerName="revoke-keys" Oct 30 01:13:59.853917 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.853917 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67717cb8-67ca-422d-ac77-77be7d572941" containerName="revoke-keys" Oct 30 01:13:59.854125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.854111 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="67717cb8-67ca-422d-ac77-77be7d572941" containerName="revoke-keys" Oct 30 01:13:59.857953 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.857925 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:13:59.871982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.871951 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:13:59.878865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.878829 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws"] Oct 30 01:13:59.882945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.882912 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:13:59.902063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.902018 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46308dd8-n65ws\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:13:59.902294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.902122 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qdr\" (UniqueName: \"kubernetes.io/projected/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-kube-api-access-v2qdr\") pod \"maas-api-revoke-sub-46308dd8-n65ws\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:13:59.902391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:13:59.902368 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46308dd8-n65ws\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:00.004429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.004378 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46308dd8-n65ws\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:00.004624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.004471 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qdr\" (UniqueName: \"kubernetes.io/projected/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-kube-api-access-v2qdr\") pod \"maas-api-revoke-sub-46308dd8-n65ws\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:00.004773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.004686 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46308dd8-n65ws\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:00.005550 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.005523 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46308dd8-n65ws\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:00.007943 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.007913 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46308dd8-n65ws\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:00.027628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.027589 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qdr\" (UniqueName: \"kubernetes.io/projected/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-kube-api-access-v2qdr\") pod \"maas-api-revoke-sub-46308dd8-n65ws\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:00.176651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.176558 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:00.378050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.378014 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:00.512669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.512631 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws"] Oct 30 01:14:00.513530 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:14:00.513499 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaf3f59_b4c7_4a15_bf00_216d8a3e0a9a.slice/crio-a3adaa359bf8f6449d0c15772f81c6a811432b9ab83a8272aec9f80a04fecc65 WatchSource:0}: Error finding container a3adaa359bf8f6449d0c15772f81c6a811432b9ab83a8272aec9f80a04fecc65: Status 404 returned error can't find the container with id a3adaa359bf8f6449d0c15772f81c6a811432b9ab83a8272aec9f80a04fecc65 Oct 30 01:14:00.711428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:00.711380 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" event={"ID":"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a","Type":"ContainerStarted","Data":"a3adaa359bf8f6449d0c15772f81c6a811432b9ab83a8272aec9f80a04fecc65"} Oct 30 01:14:00.763845 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:14:00.763754 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:14:01.720980 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:01.720946 4517 generic.go:356] "Generic (PLEG): container finished" podID="4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a" containerID="aaefc81f7f4dec91bd35ea8931ddfa86797db35acc574bf73986edc93ed0c3d5" exitCode=0 Oct 30 01:14:01.721381 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:01.720999 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" event={"ID":"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a","Type":"ContainerDied","Data":"aaefc81f7f4dec91bd35ea8931ddfa86797db35acc574bf73986edc93ed0c3d5"} Oct 30 01:14:02.046333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:02.046296 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:02.921489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:02.921445 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:03.073290 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.073257 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-openshift-ca\") pod \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " Oct 30 01:14:03.073505 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.073344 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-kube-api-access-v2qdr\" (UniqueName: \"kubernetes.io/projected/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-kube-api-access-v2qdr\") pod \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " Oct 30 01:14:03.073609 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.073581 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-xks-ca\") pod \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\" (UID: \"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a\") " Oct 30 01:14:03.073723 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.073677 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-openshift-ca" pod "4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a" (UID: "4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:03.074181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.074159 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:03.076493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.076463 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-xks-ca" pod "4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a" (UID: "4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:14:03.076617 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.076495 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-kube-api-access-v2qdr" pod "4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a" (UID: "4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a"). InnerVolumeSpecName "kube-api-access-v2qdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:03.175463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.175426 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:03.175463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.175458 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2qdr\" (UniqueName: \"kubernetes.io/projected/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a-kube-api-access-v2qdr\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:03.636249 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.636213 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:03.749133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.747988 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" event={"ID":"4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a","Type":"ContainerDied","Data":"a3adaa359bf8f6449d0c15772f81c6a811432b9ab83a8272aec9f80a04fecc65"} Oct 30 01:14:03.749133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.748107 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3adaa359bf8f6449d0c15772f81c6a811432b9ab83a8272aec9f80a04fecc65" Oct 30 01:14:03.749133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:03.748189 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws" Oct 30 01:14:05.425047 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:05.424997 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:07.064085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:07.064042 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:08.669574 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:08.669538 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:10.317621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.317582 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx"] Oct 30 01:14:10.318337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.318319 4517 cpu_manager.go:414] "containerMap: removing container" podUID="4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a" containerName="revoke-keys" Oct 30 01:14:10.318337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.318338 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a" containerName="revoke-keys" Oct 30 01:14:10.318473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.318462 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a" containerName="revoke-keys" Oct 30 01:14:10.322402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.322379 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.341136 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.341095 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx"] Oct 30 01:14:10.345905 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.345870 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:14:10.346847 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.346821 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:14:10.397569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.397522 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e7a0a6-7hqgx\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.398079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.398057 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mnbr\" (UniqueName: \"kubernetes.io/projected/9c51e7f8-ead6-4b99-be17-d179093cd9f6-kube-api-access-6mnbr\") pod \"maas-api-revoke-sub-f1e7a0a6-7hqgx\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.398337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.398318 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e7a0a6-7hqgx\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.418506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.418473 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:10.508753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.508684 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e7a0a6-7hqgx\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.510378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.510347 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mnbr\" (UniqueName: \"kubernetes.io/projected/9c51e7f8-ead6-4b99-be17-d179093cd9f6-kube-api-access-6mnbr\") pod \"maas-api-revoke-sub-f1e7a0a6-7hqgx\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.510730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.510693 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e7a0a6-7hqgx\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.518210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.518171 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e7a0a6-7hqgx\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.521412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.521378 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e7a0a6-7hqgx\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.542595 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.542563 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mnbr\" (UniqueName: \"kubernetes.io/projected/9c51e7f8-ead6-4b99-be17-d179093cd9f6-kube-api-access-6mnbr\") pod \"maas-api-revoke-sub-f1e7a0a6-7hqgx\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.637961 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.637866 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:10.956969 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:10.956940 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx"] Oct 30 01:14:11.824597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:11.824553 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" event={"ID":"9c51e7f8-ead6-4b99-be17-d179093cd9f6","Type":"ContainerStarted","Data":"a60558906294521250a268f2687fe2e7f200a163812fea432692f563ac62c0c0"} Oct 30 01:14:11.825140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:11.824618 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" event={"ID":"9c51e7f8-ead6-4b99-be17-d179093cd9f6","Type":"ContainerStarted","Data":"6f638c578be023595b816f0b96a8d89b0073fed053dc925a9f8d27f737054458"} Oct 30 01:14:11.866888 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:11.866812 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" podStartSLOduration=1.866788589 podStartE2EDuration="1.866788589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:11.861131809 +0000 UTC m=+2389.773258965" watchObservedRunningTime="2026-10-30 01:14:11.866788589 +0000 UTC m=+2389.778915731" Oct 30 01:14:12.092484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:12.092431 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:12.271735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:12.270246 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:12.834209 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:12.834170 4517 generic.go:356] "Generic (PLEG): container finished" podID="9c51e7f8-ead6-4b99-be17-d179093cd9f6" containerID="a60558906294521250a268f2687fe2e7f200a163812fea432692f563ac62c0c0" exitCode=0 Oct 30 01:14:12.834662 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:12.834247 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" event={"ID":"9c51e7f8-ead6-4b99-be17-d179093cd9f6","Type":"ContainerDied","Data":"a60558906294521250a268f2687fe2e7f200a163812fea432692f563ac62c0c0"} Oct 30 01:14:13.822398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:13.822365 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:14.114551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.114511 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:14.125152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.125121 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:14.252507 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.251295 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-xks-ca\") pod \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " Oct 30 01:14:14.252507 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.251382 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c51e7f8-ead6-4b99-be17-d179093cd9f6-kube-api-access-6mnbr\" (UniqueName: \"kubernetes.io/projected/9c51e7f8-ead6-4b99-be17-d179093cd9f6-kube-api-access-6mnbr\") pod \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " Oct 30 01:14:14.252507 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.251607 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-openshift-ca\") pod \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\" (UID: \"9c51e7f8-ead6-4b99-be17-d179093cd9f6\") " Oct 30 01:14:14.281730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.273047 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-openshift-ca" pod "9c51e7f8-ead6-4b99-be17-d179093cd9f6" (UID: "9c51e7f8-ead6-4b99-be17-d179093cd9f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:14.300366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.300306 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-xks-ca" pod "9c51e7f8-ead6-4b99-be17-d179093cd9f6" (UID: "9c51e7f8-ead6-4b99-be17-d179093cd9f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:14:14.338807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.329024 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c51e7f8-ead6-4b99-be17-d179093cd9f6-kube-api-access-6mnbr" pod "9c51e7f8-ead6-4b99-be17-d179093cd9f6" (UID: "9c51e7f8-ead6-4b99-be17-d179093cd9f6"). InnerVolumeSpecName "kube-api-access-6mnbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:14.390148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.363087 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:14.390148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.363128 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mnbr\" (UniqueName: \"kubernetes.io/projected/9c51e7f8-ead6-4b99-be17-d179093cd9f6-kube-api-access-6mnbr\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:14.390148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.363142 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51e7f8-ead6-4b99-be17-d179093cd9f6-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:14.781013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.780980 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:14.851262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.851229 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" Oct 30 01:14:14.851262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.851242 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx" event={"ID":"9c51e7f8-ead6-4b99-be17-d179093cd9f6","Type":"ContainerDied","Data":"6f638c578be023595b816f0b96a8d89b0073fed053dc925a9f8d27f737054458"} Oct 30 01:14:14.851524 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:14.851294 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f638c578be023595b816f0b96a8d89b0073fed053dc925a9f8d27f737054458" Oct 30 01:14:15.233973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:15.233911 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:15.811516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:15.811475 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:16.150941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:16.150842 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:17.428036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:17.427993 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:17.822137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:17.822102 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:18.031292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:18.031255 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:18.149972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:18.149894 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:18.950474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:18.950436 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:19.064522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:19.064486 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:19.832791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:19.832754 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:20.603358 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.603316 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:20.783987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.783941 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm"] Oct 30 01:14:20.784635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.784617 4517 cpu_manager.go:414] "containerMap: removing container" podUID="9c51e7f8-ead6-4b99-be17-d179093cd9f6" containerName="revoke-keys" Oct 30 01:14:20.784635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.784634 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c51e7f8-ead6-4b99-be17-d179093cd9f6" containerName="revoke-keys" Oct 30 01:14:20.784829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.784755 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c51e7f8-ead6-4b99-be17-d179093cd9f6" containerName="revoke-keys" Oct 30 01:14:20.788664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.788637 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:20.806606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.806569 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm"] Oct 30 01:14:20.810968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.810913 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-481249\"" Oct 30 01:14:20.816482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.816433 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-481249-dockercfg-pdpzc\"" Oct 30 01:14:20.820493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.820062 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm"] Oct 30 01:14:20.820493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.820213 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:20.836195 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.835791 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm"] Oct 30 01:14:20.943792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.943167 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm\" (UID: \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:20.943792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.943423 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k482d\" (UniqueName: \"kubernetes.io/projected/be6f389c-edee-4033-ab14-1aec373be293-kube-api-access-k482d\") pod \"payload-processing-e2e-ipp-a-481249-65677df64-8bpkm\" (UID: \"be6f389c-edee-4033-ab14-1aec373be293\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:20.943792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.943598 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6f389c-edee-4033-ab14-1aec373be293-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-481249-65677df64-8bpkm\" (UID: \"be6f389c-edee-4033-ab14-1aec373be293\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:20.943792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:20.943645 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dpfj\" (UniqueName: \"kubernetes.io/projected/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-kube-api-access-5dpfj\") pod \"payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm\" (UID: \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:21.083825 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.073694 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k482d\" (UniqueName: \"kubernetes.io/projected/be6f389c-edee-4033-ab14-1aec373be293-kube-api-access-k482d\") pod \"payload-processing-e2e-ipp-a-481249-65677df64-8bpkm\" (UID: \"be6f389c-edee-4033-ab14-1aec373be293\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:21.083825 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.073922 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6f389c-edee-4033-ab14-1aec373be293-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-481249-65677df64-8bpkm\" (UID: \"be6f389c-edee-4033-ab14-1aec373be293\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:21.083825 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.073965 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dpfj\" (UniqueName: \"kubernetes.io/projected/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-kube-api-access-5dpfj\") pod \"payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm\" (UID: \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:21.083825 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.074018 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm\" (UID: \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:21.083825 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.074994 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6f389c-edee-4033-ab14-1aec373be293-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-481249-65677df64-8bpkm\" (UID: \"be6f389c-edee-4033-ab14-1aec373be293\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:21.083825 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.075168 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm\" (UID: \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:21.163940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.160498 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k482d\" (UniqueName: \"kubernetes.io/projected/be6f389c-edee-4033-ab14-1aec373be293-kube-api-access-k482d\") pod \"payload-processing-e2e-ipp-a-481249-65677df64-8bpkm\" (UID: \"be6f389c-edee-4033-ab14-1aec373be293\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:21.182736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.181486 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dpfj\" (UniqueName: \"kubernetes.io/projected/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-kube-api-access-5dpfj\") pod \"payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm\" (UID: \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:21.221809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.220458 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:21.403491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.403449 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:21.424205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.424165 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:21.651517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.651457 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm"] Oct 30 01:14:21.653030 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:14:21.652996 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3136a9e_bba3_4a76_b3cf_4fc2d85eb025.slice/crio-c038457f863f970a9a127baa8284c78445316575fded2721671c6bd09979057b WatchSource:0}: Error finding container c038457f863f970a9a127baa8284c78445316575fded2721671c6bd09979057b: Status 404 returned error can't find the container with id c038457f863f970a9a127baa8284c78445316575fded2721671c6bd09979057b Oct 30 01:14:21.784426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.784387 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm"] Oct 30 01:14:21.784995 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:14:21.784967 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6f389c_edee_4033_ab14_1aec373be293.slice/crio-98f7b3df62023153ac5bec3a6a8574a19398a1865d65890ada0994e8aada96d9 WatchSource:0}: Error finding container 98f7b3df62023153ac5bec3a6a8574a19398a1865d65890ada0994e8aada96d9: Status 404 returned error can't find the container with id 98f7b3df62023153ac5bec3a6a8574a19398a1865d65890ada0994e8aada96d9 Oct 30 01:14:21.901862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:21.901831 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:22.266937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:22.266873 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" event={"ID":"be6f389c-edee-4033-ab14-1aec373be293","Type":"ContainerStarted","Data":"d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da"} Oct 30 01:14:22.267137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:22.266953 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:22.267137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:22.266981 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" event={"ID":"be6f389c-edee-4033-ab14-1aec373be293","Type":"ContainerStarted","Data":"98f7b3df62023153ac5bec3a6a8574a19398a1865d65890ada0994e8aada96d9"} Oct 30 01:14:22.268945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:22.268910 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" event={"ID":"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025","Type":"ContainerStarted","Data":"cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac"} Oct 30 01:14:22.269133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:22.268969 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" event={"ID":"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025","Type":"ContainerStarted","Data":"c038457f863f970a9a127baa8284c78445316575fded2721671c6bd09979057b"} Oct 30 01:14:22.269133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:22.268989 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:22.306722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:22.306649 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" podStartSLOduration=2.306632247 podStartE2EDuration="2.306632247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:22.303063803 +0000 UTC m=+2400.215190946" watchObservedRunningTime="2026-10-30 01:14:22.306632247 +0000 UTC m=+2400.218759388" Oct 30 01:14:22.340259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:22.340186 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" podStartSLOduration=2.340164137 podStartE2EDuration="2.340164137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:22.338934089 +0000 UTC m=+2400.251061230" watchObservedRunningTime="2026-10-30 01:14:22.340164137 +0000 UTC m=+2400.252291283" Oct 30 01:14:23.730734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:23.711201 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:23.946735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:23.913274 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:24.340737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:24.337268 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:24.933685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:24.933510 4517 scope.go:122] "RemoveContainer" containerID="832b5b4c4d266606c925136d4671b5880ef2a88857c0b19ed9a7fde0b797ccbb" Oct 30 01:14:25.590778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:25.572773 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:25.827734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:25.813581 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:25.827734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:25.813950 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 01:14:25.971806 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:25.965750 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.025755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.012842 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.045749 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.043124 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.076500 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.075771 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.077286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.077252 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.232053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.231944 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.384742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.380873 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:14:26.458305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.458247 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.474723 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.473864 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" podUID="078c1897-20b9-49d7-8096-d60f593e419e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.159:8000/ready\": context deadline exceeded" Oct 30 01:14:26.692961 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.692766 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.695757 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.694963 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 01:14:26.765473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:26.765437 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 01:14:27.155040 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:27.154995 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:28.899205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:28.899165 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8985-f57532"] Oct 30 01:14:28.905188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:28.905160 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8985-f57532" Oct 30 01:14:28.920516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:28.920485 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8985-f57532"] Oct 30 01:14:28.925748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:28.924488 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v4hbx\"" Oct 30 01:14:28.993538 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:28.993487 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-486vg\" (UniqueName: \"kubernetes.io/projected/b2872324-64d6-43c4-995f-96e6a147f76f-kube-api-access-486vg\") pod \"test-curl-8985-f57532\" (UID: \"b2872324-64d6-43c4-995f-96e6a147f76f\") " pod="opendatahub/test-curl-8985-f57532" Oct 30 01:14:29.094899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:29.094852 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486vg\" (UniqueName: \"kubernetes.io/projected/b2872324-64d6-43c4-995f-96e6a147f76f-kube-api-access-486vg\") pod \"test-curl-8985-f57532\" (UID: \"b2872324-64d6-43c4-995f-96e6a147f76f\") " pod="opendatahub/test-curl-8985-f57532" Oct 30 01:14:29.116671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:29.116632 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-486vg\" (UniqueName: \"kubernetes.io/projected/b2872324-64d6-43c4-995f-96e6a147f76f-kube-api-access-486vg\") pod \"test-curl-8985-f57532\" (UID: \"b2872324-64d6-43c4-995f-96e6a147f76f\") " pod="opendatahub/test-curl-8985-f57532" Oct 30 01:14:29.122169 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:29.122129 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:29.222555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:29.222194 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8985-f57532" Oct 30 01:14:29.544583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:29.544537 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8985-f57532"] Oct 30 01:14:29.553687 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:14:29.553651 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2872324_64d6_43c4_995f_96e6a147f76f.slice/crio-e09a2e88c88a6138f40b7667665f5e5386df7adcf8d47dd6f881dc04c89b7163 WatchSource:0}: Error finding container e09a2e88c88a6138f40b7667665f5e5386df7adcf8d47dd6f881dc04c89b7163: Status 404 returned error can't find the container with id e09a2e88c88a6138f40b7667665f5e5386df7adcf8d47dd6f881dc04c89b7163 Oct 30 01:14:29.729099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:29.729063 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8985-f57532" event={"ID":"b2872324-64d6-43c4-995f-96e6a147f76f","Type":"ContainerStarted","Data":"b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5"} Oct 30 01:14:29.729317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:29.729107 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8985-f57532" event={"ID":"b2872324-64d6-43c4-995f-96e6a147f76f","Type":"ContainerStarted","Data":"e09a2e88c88a6138f40b7667665f5e5386df7adcf8d47dd6f881dc04c89b7163"} Oct 30 01:14:29.765562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:29.765506 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8985-f57532" podStartSLOduration=1.765490077 podStartE2EDuration="1.765490077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:29.762656476 +0000 UTC m=+2407.674783617" watchObservedRunningTime="2026-10-30 01:14:29.765490077 +0000 UTC m=+2407.677617218" Oct 30 01:14:30.620660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:30.620606 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8985-f57532"] Oct 30 01:14:30.625514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:30.625481 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8985-f57532"] Oct 30 01:14:30.766327 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:14:30.766278 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:14:30.958794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:30.957072 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z"] Oct 30 01:14:30.958794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:30.958199 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b2872324-64d6-43c4-995f-96e6a147f76f" containerName="test-curl-8985-f57532" Oct 30 01:14:30.958794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:30.958225 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2872324-64d6-43c4-995f-96e6a147f76f" containerName="test-curl-8985-f57532" Oct 30 01:14:30.958794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:30.958369 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2872324-64d6-43c4-995f-96e6a147f76f" containerName="test-curl-8985-f57532" Oct 30 01:14:30.981995 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:30.981780 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:30.991752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:30.991594 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z"] Oct 30 01:14:30.991752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:30.991754 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.008313 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.007984 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:14:31.008313 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.008149 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:14:31.033103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.033069 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20155e60-nxg2z\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.033292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.033269 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7z7g\" (UniqueName: \"kubernetes.io/projected/d3770123-5dcb-4bb5-b671-34010ea73246-kube-api-access-v7z7g\") pod \"maas-api-revoke-sub-20155e60-nxg2z\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.033415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.033399 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20155e60-nxg2z\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.135159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.135114 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7z7g\" (UniqueName: \"kubernetes.io/projected/d3770123-5dcb-4bb5-b671-34010ea73246-kube-api-access-v7z7g\") pod \"maas-api-revoke-sub-20155e60-nxg2z\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.135382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.135270 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20155e60-nxg2z\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.135465 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.135422 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20155e60-nxg2z\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.136100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.136071 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20155e60-nxg2z\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.138428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.138395 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20155e60-nxg2z\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.157270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.157234 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7z7g\" (UniqueName: \"kubernetes.io/projected/d3770123-5dcb-4bb5-b671-34010ea73246-kube-api-access-v7z7g\") pod \"maas-api-revoke-sub-20155e60-nxg2z\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.317101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.317072 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:31.629738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.629695 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z"] Oct 30 01:14:31.632086 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:14:31.632053 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3770123_5dcb_4bb5_b671_34010ea73246.slice/crio-c88c93de25b44f14bcb6a664192d458b0199443b74e153e5f1bc60079e32a682 WatchSource:0}: Error finding container c88c93de25b44f14bcb6a664192d458b0199443b74e153e5f1bc60079e32a682: Status 404 returned error can't find the container with id c88c93de25b44f14bcb6a664192d458b0199443b74e153e5f1bc60079e32a682 Oct 30 01:14:31.747018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.746984 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" event={"ID":"d3770123-5dcb-4bb5-b671-34010ea73246","Type":"ContainerStarted","Data":"c88c93de25b44f14bcb6a664192d458b0199443b74e153e5f1bc60079e32a682"} Oct 30 01:14:31.747248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:31.747104 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8985-f57532" podUID="b2872324-64d6-43c4-995f-96e6a147f76f" containerName="test-curl-8985-f57532" containerID="cri-o://b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5" gracePeriod=2 Oct 30 01:14:31.759210 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:14:31.759167 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8985-f57532\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="b2872324-64d6-43c4-995f-96e6a147f76f" pod="opendatahub/test-curl-8985-f57532" Oct 30 01:14:32.243810 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:14:32.243761 4517 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8985-f57532\" is forbidden: User \"system:node:ip-10-0-132-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-84.ec2.internal' and this object" podUID="b2872324-64d6-43c4-995f-96e6a147f76f" pod="opendatahub/test-curl-8985-f57532" Oct 30 01:14:32.507391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:32.507283 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:32.757555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:32.757468 4517 generic.go:356] "Generic (PLEG): container finished" podID="d3770123-5dcb-4bb5-b671-34010ea73246" containerID="eeadfe4d3331ccf76f0fa5142cb95fe73361adfcfbbe7643f692e15601d9a1b3" exitCode=0 Oct 30 01:14:32.757971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:32.757549 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" event={"ID":"d3770123-5dcb-4bb5-b671-34010ea73246","Type":"ContainerDied","Data":"eeadfe4d3331ccf76f0fa5142cb95fe73361adfcfbbe7643f692e15601d9a1b3"} Oct 30 01:14:33.284027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:33.283993 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:14:33.284205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:33.284085 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:14:33.979344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:33.979314 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:34.035726 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.035678 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8985-f57532" Oct 30 01:14:34.125200 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.125162 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3770123-5dcb-4bb5-b671-34010ea73246-kube-api-access-v7z7g\" (UniqueName: \"kubernetes.io/projected/d3770123-5dcb-4bb5-b671-34010ea73246-kube-api-access-v7z7g\") pod \"d3770123-5dcb-4bb5-b671-34010ea73246\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " Oct 30 01:14:34.125380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.125241 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2872324-64d6-43c4-995f-96e6a147f76f-kube-api-access-486vg\" (UniqueName: \"kubernetes.io/projected/b2872324-64d6-43c4-995f-96e6a147f76f-kube-api-access-486vg\") pod \"b2872324-64d6-43c4-995f-96e6a147f76f\" (UID: \"b2872324-64d6-43c4-995f-96e6a147f76f\") " Oct 30 01:14:34.125380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.125377 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-xks-ca\") pod \"d3770123-5dcb-4bb5-b671-34010ea73246\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " Oct 30 01:14:34.125517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.125448 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-openshift-ca\") pod \"d3770123-5dcb-4bb5-b671-34010ea73246\" (UID: \"d3770123-5dcb-4bb5-b671-34010ea73246\") " Oct 30 01:14:34.125901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.125872 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-openshift-ca" pod "d3770123-5dcb-4bb5-b671-34010ea73246" (UID: "d3770123-5dcb-4bb5-b671-34010ea73246"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:34.126188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.126164 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:34.127851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.127826 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3770123-5dcb-4bb5-b671-34010ea73246-kube-api-access-v7z7g" pod "d3770123-5dcb-4bb5-b671-34010ea73246" (UID: "d3770123-5dcb-4bb5-b671-34010ea73246"). InnerVolumeSpecName "kube-api-access-v7z7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:34.127936 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.127865 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2872324-64d6-43c4-995f-96e6a147f76f-kube-api-access-486vg" pod "b2872324-64d6-43c4-995f-96e6a147f76f" (UID: "b2872324-64d6-43c4-995f-96e6a147f76f"). InnerVolumeSpecName "kube-api-access-486vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:34.127984 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.127930 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-xks-ca" pod "d3770123-5dcb-4bb5-b671-34010ea73246" (UID: "d3770123-5dcb-4bb5-b671-34010ea73246"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:14:34.227999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.227958 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7z7g\" (UniqueName: \"kubernetes.io/projected/d3770123-5dcb-4bb5-b671-34010ea73246-kube-api-access-v7z7g\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:34.227999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.228000 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-486vg\" (UniqueName: \"kubernetes.io/projected/b2872324-64d6-43c4-995f-96e6a147f76f-kube-api-access-486vg\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:34.228285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.228015 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3770123-5dcb-4bb5-b671-34010ea73246-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:34.242295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.242257 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2872324-64d6-43c4-995f-96e6a147f76f" path="/var/lib/kubelet/pods/b2872324-64d6-43c4-995f-96e6a147f76f/volumes" Oct 30 01:14:34.320496 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.320465 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:34.780459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.780423 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" Oct 30 01:14:34.780665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.780420 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z" event={"ID":"d3770123-5dcb-4bb5-b671-34010ea73246","Type":"ContainerDied","Data":"c88c93de25b44f14bcb6a664192d458b0199443b74e153e5f1bc60079e32a682"} Oct 30 01:14:34.780665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.780556 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c88c93de25b44f14bcb6a664192d458b0199443b74e153e5f1bc60079e32a682" Oct 30 01:14:34.782213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.782178 4517 generic.go:356] "Generic (PLEG): container finished" podID="b2872324-64d6-43c4-995f-96e6a147f76f" containerID="b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5" exitCode=137 Oct 30 01:14:34.782355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.782228 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8985-f57532" Oct 30 01:14:34.782542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.782488 4517 scope.go:122] "RemoveContainer" containerID="b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5" Oct 30 01:14:34.796621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.796594 4517 scope.go:122] "RemoveContainer" containerID="b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5" Oct 30 01:14:34.797091 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:14:34.797065 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5\": container with ID starting with b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5 not found: ID does not exist" containerID="b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5" Oct 30 01:14:34.797169 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:34.797100 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5"} err="failed to get container status \"b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5\": rpc error: code = NotFound desc = could not find container \"b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5\": container with ID starting with b8b7f1a13e756d4df417e1aa77a5a05c33c55dc45b2ff7ce75b92b9f51d159e5 not found: ID does not exist" Oct 30 01:14:35.993163 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:35.993114 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:37.633438 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:37.633402 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:39.424135 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:39.424092 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:41.104941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.104904 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:41.360412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.360307 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf"] Oct 30 01:14:41.361412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.361386 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d3770123-5dcb-4bb5-b671-34010ea73246" containerName="revoke-keys" Oct 30 01:14:41.361412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.361413 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3770123-5dcb-4bb5-b671-34010ea73246" containerName="revoke-keys" Oct 30 01:14:41.361580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.361567 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3770123-5dcb-4bb5-b671-34010ea73246" containerName="revoke-keys" Oct 30 01:14:41.367259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.367228 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.386131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.386093 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf"] Oct 30 01:14:41.476332 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.476286 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:14:41.476558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.476388 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:14:41.552299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.552261 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv4rn\" (UniqueName: \"kubernetes.io/projected/19c06538-337b-41cc-ac02-e3eab004c9de-kube-api-access-tv4rn\") pod \"maas-api-revoke-sub-48e13d8d-68lrf\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.552538 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.552344 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e13d8d-68lrf\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.552611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.552543 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e13d8d-68lrf\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.654139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.654033 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4rn\" (UniqueName: \"kubernetes.io/projected/19c06538-337b-41cc-ac02-e3eab004c9de-kube-api-access-tv4rn\") pod \"maas-api-revoke-sub-48e13d8d-68lrf\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.654334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.654150 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e13d8d-68lrf\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.654334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.654292 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e13d8d-68lrf\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.654994 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.654967 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e13d8d-68lrf\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.657024 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.656998 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e13d8d-68lrf\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.676976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.676937 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4rn\" (UniqueName: \"kubernetes.io/projected/19c06538-337b-41cc-ac02-e3eab004c9de-kube-api-access-tv4rn\") pod \"maas-api-revoke-sub-48e13d8d-68lrf\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.688787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.688694 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:41.810512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:41.810394 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:42.126978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.124121 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf"] Oct 30 01:14:42.351416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.351321 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:42.790927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.790887 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h"] Oct 30 01:14:42.795823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.795793 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.811728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.809011 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-ea2547-openshift-default-dockercfg-g9jmp\"" Oct 30 01:14:42.820872 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.820827 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-podinfo\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.821099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.821037 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.821225 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.821103 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-token\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.821225 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.821152 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-data\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.821336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.821264 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-credential-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.821336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.821302 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-certs\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.821449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.821342 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-envoy\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.821554 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.821518 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istiod-ca-cert\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.821686 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.821602 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrk4l\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-kube-api-access-hrk4l\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.829800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.829759 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h"] Oct 30 01:14:42.864178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.864118 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" event={"ID":"19c06538-337b-41cc-ac02-e3eab004c9de","Type":"ContainerStarted","Data":"a8e32d5ee5c0495959af19e2f0b3a6025736d180e9510b3dd2541c0862903ed8"} Oct 30 01:14:42.864515 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.864475 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" event={"ID":"19c06538-337b-41cc-ac02-e3eab004c9de","Type":"ContainerStarted","Data":"c2ecd10495de694bb9fe555430f5b8f09bd60c56c491f974cfc14c10e1c5e402"} Oct 30 01:14:42.908417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.908349 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" podStartSLOduration=1.908332015 podStartE2EDuration="1.908332015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:42.905487115 +0000 UTC m=+2420.817614258" watchObservedRunningTime="2026-10-30 01:14:42.908332015 +0000 UTC m=+2420.820459155" Oct 30 01:14:42.923494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.923386 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-podinfo\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.923698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.923578 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.923841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.923782 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-token\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.923903 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.923840 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-data\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.923960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.923913 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-credential-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.923960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.923954 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-certs\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.924072 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.923998 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-envoy\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.924125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.924117 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istiod-ca-cert\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.924238 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.924199 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrk4l\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-kube-api-access-hrk4l\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.924378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.924350 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-data\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.924791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.924762 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.925133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.925078 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-credential-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.925133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.925128 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-certs\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.925809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.925784 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istiod-ca-cert\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.926860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.926834 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-podinfo\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.927086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.927062 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-envoy\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.929859 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.929832 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:42.946510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.946469 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-token\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:42.946722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:42.946572 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrk4l\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-kube-api-access-hrk4l\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:43.116056 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.116014 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:43.459993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.459882 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h"] Oct 30 01:14:43.465384 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:14:43.465349 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3223b0_eab8_4a4f_ba99_ddd95c3da3de.slice/crio-9c6b4cba8edbb588f5fbdbe415559ba6b49d6cc2f7353ac021b8774535b8c1f8 WatchSource:0}: Error finding container 9c6b4cba8edbb588f5fbdbe415559ba6b49d6cc2f7353ac021b8774535b8c1f8: Status 404 returned error can't find the container with id 9c6b4cba8edbb588f5fbdbe415559ba6b49d6cc2f7353ac021b8774535b8c1f8 Oct 30 01:14:43.469599 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.469555 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:14:43.469804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.469660 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:14:43.469804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.469724 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:14:43.874975 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.874932 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" event={"ID":"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de","Type":"ContainerStarted","Data":"305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16"} Oct 30 01:14:43.875213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.875003 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" event={"ID":"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de","Type":"ContainerStarted","Data":"9c6b4cba8edbb588f5fbdbe415559ba6b49d6cc2f7353ac021b8774535b8c1f8"} Oct 30 01:14:43.876911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.876879 4517 generic.go:356] "Generic (PLEG): container finished" podID="19c06538-337b-41cc-ac02-e3eab004c9de" containerID="a8e32d5ee5c0495959af19e2f0b3a6025736d180e9510b3dd2541c0862903ed8" exitCode=0 Oct 30 01:14:43.877072 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.876971 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" event={"ID":"19c06538-337b-41cc-ac02-e3eab004c9de","Type":"ContainerDied","Data":"a8e32d5ee5c0495959af19e2f0b3a6025736d180e9510b3dd2541c0862903ed8"} Oct 30 01:14:43.920802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:43.920740 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podStartSLOduration=1.9207227599999999 podStartE2EDuration="1.92072276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:43.917427363 +0000 UTC m=+2421.829554508" watchObservedRunningTime="2026-10-30 01:14:43.92072276 +0000 UTC m=+2421.832849891" Oct 30 01:14:44.116258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:44.116208 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:44.118360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:44.118328 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:14:44.118533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:44.118386 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 01:14:44.711724 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:44.711676 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:45.116443 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.116406 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:14:45.116583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.116459 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 01:14:45.150581 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.150550 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:45.299907 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.299860 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c06538-337b-41cc-ac02-e3eab004c9de-kube-api-access-tv4rn\" (UniqueName: \"kubernetes.io/projected/19c06538-337b-41cc-ac02-e3eab004c9de-kube-api-access-tv4rn\") pod \"19c06538-337b-41cc-ac02-e3eab004c9de\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " Oct 30 01:14:45.300092 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.299959 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-openshift-ca\") pod \"19c06538-337b-41cc-ac02-e3eab004c9de\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " Oct 30 01:14:45.300092 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.300018 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-xks-ca\") pod \"19c06538-337b-41cc-ac02-e3eab004c9de\" (UID: \"19c06538-337b-41cc-ac02-e3eab004c9de\") " Oct 30 01:14:45.300340 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.300313 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-openshift-ca" pod "19c06538-337b-41cc-ac02-e3eab004c9de" (UID: "19c06538-337b-41cc-ac02-e3eab004c9de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:45.300839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.300814 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.302365 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.302334 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-xks-ca" pod "19c06538-337b-41cc-ac02-e3eab004c9de" (UID: "19c06538-337b-41cc-ac02-e3eab004c9de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:14:45.302464 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.302413 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c06538-337b-41cc-ac02-e3eab004c9de-kube-api-access-tv4rn" pod "19c06538-337b-41cc-ac02-e3eab004c9de" (UID: "19c06538-337b-41cc-ac02-e3eab004c9de"). InnerVolumeSpecName "kube-api-access-tv4rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:45.403084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.403039 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv4rn\" (UniqueName: \"kubernetes.io/projected/19c06538-337b-41cc-ac02-e3eab004c9de-kube-api-access-tv4rn\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.403084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.403083 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c06538-337b-41cc-ac02-e3eab004c9de-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:14:45.919741 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.919693 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" Oct 30 01:14:45.920283 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.919740 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf" event={"ID":"19c06538-337b-41cc-ac02-e3eab004c9de","Type":"ContainerDied","Data":"c2ecd10495de694bb9fe555430f5b8f09bd60c56c491f974cfc14c10e1c5e402"} Oct 30 01:14:45.920283 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:45.919795 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ecd10495de694bb9fe555430f5b8f09bd60c56c491f974cfc14c10e1c5e402" Oct 30 01:14:46.116743 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:46.116694 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:14:46.116943 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:46.116771 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 01:14:46.412634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:46.412575 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:47.117468 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:47.117426 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:14:47.118084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:47.117497 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 01:14:48.104319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:48.104280 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:48.116606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:48.116561 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:14:48.116864 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:48.116619 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 01:14:49.116833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:49.116773 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:14:49.117390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:49.116851 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 01:14:49.687394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:49.687339 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:50.117479 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:50.117423 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 01:14:50.118054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:50.117505 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 01:14:50.417569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:50.417473 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:51.029563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:51.029521 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:51.123195 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:51.123160 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:51.123673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:51.123651 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:51.124755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:51.124699 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:14:51.503254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:51.503215 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:51.667627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:51.667590 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:52.234402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.229437 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:52.325682 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.325648 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx"] Oct 30 01:14:52.326348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.326331 4517 cpu_manager.go:414] "containerMap: removing container" podUID="19c06538-337b-41cc-ac02-e3eab004c9de" containerName="revoke-keys" Oct 30 01:14:52.326447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.326349 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c06538-337b-41cc-ac02-e3eab004c9de" containerName="revoke-keys" Oct 30 01:14:52.326491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.326443 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c06538-337b-41cc-ac02-e3eab004c9de" containerName="revoke-keys" Oct 30 01:14:52.330233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.330209 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.344460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.344434 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-33e759-openshift-default-dockercfg-6b9m2\"" Oct 30 01:14:52.359931 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.359891 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx"] Oct 30 01:14:52.418572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.418536 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e23f2-fd01-4f9d-9c43-3b570733b19b-istiod-ca-cert\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.418796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.418620 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-certs\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.418796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.418681 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-podinfo\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.418895 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.418847 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-data\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.418985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.418970 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-envoy\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.419036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.419008 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-token\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.419091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.419052 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-credential-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.419202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.419186 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6hv\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-kube-api-access-9b6hv\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.419239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.419210 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.521122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521016 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-envoy\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.521122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521086 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-token\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.521388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521142 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-credential-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.521388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521253 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6hv\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-kube-api-access-9b6hv\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.521388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521275 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.521610 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521399 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e23f2-fd01-4f9d-9c43-3b570733b19b-istiod-ca-cert\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.521610 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521487 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-certs\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.521610 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521526 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-podinfo\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.521610 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521578 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-data\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.522029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521611 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-credential-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.522029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521851 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.522029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.521908 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-certs\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.522225 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.522025 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-data\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.522590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.522569 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e23f2-fd01-4f9d-9c43-3b570733b19b-istiod-ca-cert\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.524678 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.524653 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-envoy\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.524922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.524900 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-podinfo\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.543437 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.543399 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6hv\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-kube-api-access-9b6hv\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.544630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.544599 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-token\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.645460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.645412 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:52.978469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.978430 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx"] Oct 30 01:14:52.980088 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:14:52.980056 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870e23f2_fd01_4f9d_9c43_3b570733b19b.slice/crio-9fc7b2e25ed43b71477ecf00d81c38a701c0cdd4f101b5e52fced23bedf770f4 WatchSource:0}: Error finding container 9fc7b2e25ed43b71477ecf00d81c38a701c0cdd4f101b5e52fced23bedf770f4: Status 404 returned error can't find the container with id 9fc7b2e25ed43b71477ecf00d81c38a701c0cdd4f101b5e52fced23bedf770f4 Oct 30 01:14:52.983795 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.983756 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:14:52.983922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.983851 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:14:52.983922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:52.983902 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:14:53.014695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:53.014659 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" event={"ID":"870e23f2-fd01-4f9d-9c43-3b570733b19b","Type":"ContainerStarted","Data":"9fc7b2e25ed43b71477ecf00d81c38a701c0cdd4f101b5e52fced23bedf770f4"} Oct 30 01:14:53.417481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:53.417418 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:54.030772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:54.030723 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" event={"ID":"870e23f2-fd01-4f9d-9c43-3b570733b19b","Type":"ContainerStarted","Data":"9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda"} Oct 30 01:14:54.093734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:54.091956 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" podStartSLOduration=2.091927014 podStartE2EDuration="2.091927014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:54.084225229 +0000 UTC m=+2431.996352367" watchObservedRunningTime="2026-10-30 01:14:54.091927014 +0000 UTC m=+2432.004054157" Oct 30 01:14:54.646762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:54.646666 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:14:54.649145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:54.649115 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:14:54.649269 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:54.649187 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:14:55.210982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:55.210949 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:55.646230 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:55.646194 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:14:55.646446 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:55.646256 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:14:55.888739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:55.869805 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:55.991045 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:55.990934 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:56.646659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:56.646612 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:14:56.646923 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:56.646685 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:14:57.082733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.082457 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:57.597759 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.595962 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:57.646860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.646814 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:14:57.647047 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.646896 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:14:57.706591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.702337 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869"] Oct 30 01:14:57.715724 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.715668 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:14:57.716609 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.715512 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf"] Oct 30 01:14:57.720777 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.720568 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:57.722180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.722146 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869"] Oct 30 01:14:57.722344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.722269 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:14:57.723972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.723940 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf"] Oct 30 01:14:57.732806 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.732357 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-abf8ae\"" Oct 30 01:14:57.757393 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.757349 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcww6\" (UniqueName: \"kubernetes.io/projected/6ebe5e3d-0433-402b-82cd-19af9889bd6e-kube-api-access-tcww6\") pod \"payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf\" (UID: \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:14:57.757585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.757478 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2td2b\" (UniqueName: \"kubernetes.io/projected/8093216b-6714-405c-ab9d-d02d47e7de25-kube-api-access-2td2b\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869\" (UID: \"8093216b-6714-405c-ab9d-d02d47e7de25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:14:57.757585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.757549 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093216b-6714-405c-ab9d-d02d47e7de25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869\" (UID: \"8093216b-6714-405c-ab9d-d02d47e7de25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:14:57.780786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.757776 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebe5e3d-0433-402b-82cd-19af9889bd6e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf\" (UID: \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:14:57.881163 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.881051 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcww6\" (UniqueName: \"kubernetes.io/projected/6ebe5e3d-0433-402b-82cd-19af9889bd6e-kube-api-access-tcww6\") pod \"payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf\" (UID: \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:14:57.894995 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.894953 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2td2b\" (UniqueName: \"kubernetes.io/projected/8093216b-6714-405c-ab9d-d02d47e7de25-kube-api-access-2td2b\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869\" (UID: \"8093216b-6714-405c-ab9d-d02d47e7de25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:14:57.908383 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.908344 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093216b-6714-405c-ab9d-d02d47e7de25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869\" (UID: \"8093216b-6714-405c-ab9d-d02d47e7de25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:14:57.953266 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.925639 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebe5e3d-0433-402b-82cd-19af9889bd6e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf\" (UID: \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:14:58.051731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.909538 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093216b-6714-405c-ab9d-d02d47e7de25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869\" (UID: \"8093216b-6714-405c-ab9d-d02d47e7de25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:14:58.051731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.927507 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebe5e3d-0433-402b-82cd-19af9889bd6e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf\" (UID: \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:14:58.051731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:57.954297 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2td2b\" (UniqueName: \"kubernetes.io/projected/8093216b-6714-405c-ab9d-d02d47e7de25-kube-api-access-2td2b\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869\" (UID: \"8093216b-6714-405c-ab9d-d02d47e7de25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:14:58.078985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:58.067047 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:14:58.080565 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:58.080523 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcww6\" (UniqueName: \"kubernetes.io/projected/6ebe5e3d-0433-402b-82cd-19af9889bd6e-kube-api-access-tcww6\") pod \"payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf\" (UID: \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:14:58.110122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:58.110079 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:14:58.614623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:58.614581 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869"] Oct 30 01:14:58.616932 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:14:58.616891 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8093216b_6714_405c_ab9d_d02d47e7de25.slice/crio-a5b54a3916588b865e85557c7b8044b8ce4df037ad6c26e63c26c8a082ed1ccf WatchSource:0}: Error finding container a5b54a3916588b865e85557c7b8044b8ce4df037ad6c26e63c26c8a082ed1ccf: Status 404 returned error can't find the container with id a5b54a3916588b865e85557c7b8044b8ce4df037ad6c26e63c26c8a082ed1ccf Oct 30 01:14:58.647469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:58.647431 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:14:58.647661 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:58.647502 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:14:58.698555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:58.698520 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf"] Oct 30 01:14:58.702660 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:14:58.702617 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebe5e3d_0433_402b_82cd_19af9889bd6e.slice/crio-757ebf4caea581cd770b7dd79d8cbcd7aed2ef4bb31479461e7b5bb8678f7304 WatchSource:0}: Error finding container 757ebf4caea581cd770b7dd79d8cbcd7aed2ef4bb31479461e7b5bb8678f7304: Status 404 returned error can't find the container with id 757ebf4caea581cd770b7dd79d8cbcd7aed2ef4bb31479461e7b5bb8678f7304 Oct 30 01:14:59.022614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.022580 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:59.221214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.221104 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" event={"ID":"8093216b-6714-405c-ab9d-d02d47e7de25","Type":"ContainerStarted","Data":"027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99"} Oct 30 01:14:59.221214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.221170 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" event={"ID":"8093216b-6714-405c-ab9d-d02d47e7de25","Type":"ContainerStarted","Data":"a5b54a3916588b865e85557c7b8044b8ce4df037ad6c26e63c26c8a082ed1ccf"} Oct 30 01:14:59.221918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.221893 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:14:59.223640 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.223612 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" event={"ID":"6ebe5e3d-0433-402b-82cd-19af9889bd6e","Type":"ContainerStarted","Data":"c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7"} Oct 30 01:14:59.223762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.223661 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" event={"ID":"6ebe5e3d-0433-402b-82cd-19af9889bd6e","Type":"ContainerStarted","Data":"757ebf4caea581cd770b7dd79d8cbcd7aed2ef4bb31479461e7b5bb8678f7304"} Oct 30 01:14:59.223894 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.223879 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:14:59.290735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.287280 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" podStartSLOduration=2.287255434 podStartE2EDuration="2.287255434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:59.282928827 +0000 UTC m=+2437.195055969" watchObservedRunningTime="2026-10-30 01:14:59.287255434 +0000 UTC m=+2437.199382576" Oct 30 01:14:59.321964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.321900 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" podStartSLOduration=2.321882763 podStartE2EDuration="2.321882763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:59.319555158 +0000 UTC m=+2437.231682295" watchObservedRunningTime="2026-10-30 01:14:59.321882763 +0000 UTC m=+2437.234009935" Oct 30 01:14:59.646863 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.646822 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:14:59.647074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:14:59.646900 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:15:00.143680 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.143636 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4"] Oct 30 01:15:00.148483 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.148447 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:00.150387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.150360 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w"] Oct 30 01:15:00.155171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.155142 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:00.155551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.155527 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm"] Oct 30 01:15:00.161017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.160990 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:00.162588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.162562 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs"] Oct 30 01:15:00.164862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.164829 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:15:00.169930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.169362 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:00.171208 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.171187 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd"] Oct 30 01:15:00.183901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.180773 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw"] Oct 30 01:15:00.184177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.184153 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:00.194932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.194902 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:00.262728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.262621 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmb6x\" (UniqueName: \"kubernetes.io/projected/f34effdd-fdf2-49f3-b971-b12b0315257b-kube-api-access-bmb6x\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4\" (UID: \"f34effdd-fdf2-49f3-b971-b12b0315257b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:00.263315 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.262933 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34effdd-fdf2-49f3-b971-b12b0315257b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4\" (UID: \"f34effdd-fdf2-49f3-b971-b12b0315257b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:00.325625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.325589 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4"] Oct 30 01:15:00.325878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.325664 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm"] Oct 30 01:15:00.330845 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.328325 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w"] Oct 30 01:15:00.335426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.335394 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs"] Oct 30 01:15:00.339991 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.339959 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd"] Oct 30 01:15:00.348404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.348343 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh"] Oct 30 01:15:00.366730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.356104 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:00.366730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.361548 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw"] Oct 30 01:15:00.366730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.364788 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmb6x\" (UniqueName: \"kubernetes.io/projected/f34effdd-fdf2-49f3-b971-b12b0315257b-kube-api-access-bmb6x\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4\" (UID: \"f34effdd-fdf2-49f3-b971-b12b0315257b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:00.366730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.365238 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646cf\" (UniqueName: \"kubernetes.io/projected/d051c589-6cf9-4544-896f-5cc2d09feee3-kube-api-access-646cf\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w\" (UID: \"d051c589-6cf9-4544-896f-5cc2d09feee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:00.366730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.365381 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f77bcc-c251-49e5-a309-9dede3b87810-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs\" (UID: \"10f77bcc-c251-49e5-a309-9dede3b87810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:00.366730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.365484 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09d6932-71e7-4876-81a8-8964658c2b98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm\" (UID: \"a09d6932-71e7-4876-81a8-8964658c2b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:00.366730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.365556 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvlvb\" (UniqueName: \"kubernetes.io/projected/a09d6932-71e7-4876-81a8-8964658c2b98-kube-api-access-fvlvb\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm\" (UID: \"a09d6932-71e7-4876-81a8-8964658c2b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:00.370333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.370301 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855240f-d1f1-47c1-b8e4-d155a4ee590f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw\" (UID: \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:00.370625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.370600 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95n7l\" (UniqueName: \"kubernetes.io/projected/10f77bcc-c251-49e5-a309-9dede3b87810-kube-api-access-95n7l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs\" (UID: \"10f77bcc-c251-49e5-a309-9dede3b87810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:00.370854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.370775 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sn9c\" (UniqueName: \"kubernetes.io/projected/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-kube-api-access-5sn9c\") pod \"maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd\" (UID: \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:00.372090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.370968 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34effdd-fdf2-49f3-b971-b12b0315257b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4\" (UID: \"f34effdd-fdf2-49f3-b971-b12b0315257b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:00.372090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.371417 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5sgc\" (UniqueName: \"kubernetes.io/projected/f855240f-d1f1-47c1-b8e4-d155a4ee590f-kube-api-access-l5sgc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw\" (UID: \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:00.372090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.371484 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd\" (UID: \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:00.372090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.371572 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d051c589-6cf9-4544-896f-5cc2d09feee3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w\" (UID: \"d051c589-6cf9-4544-896f-5cc2d09feee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:00.372090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.371723 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34effdd-fdf2-49f3-b971-b12b0315257b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4\" (UID: \"f34effdd-fdf2-49f3-b971-b12b0315257b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:00.373294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.373273 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:00.373831 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.373814 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw"] Oct 30 01:15:00.380687 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.380658 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf"] Oct 30 01:15:00.388263 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.388233 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh"] Oct 30 01:15:00.388440 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.388379 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:00.390447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.390416 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmb6x\" (UniqueName: \"kubernetes.io/projected/f34effdd-fdf2-49f3-b971-b12b0315257b-kube-api-access-bmb6x\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4\" (UID: \"f34effdd-fdf2-49f3-b971-b12b0315257b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:00.391275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.391250 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw"] Oct 30 01:15:00.395886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.395807 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6"] Oct 30 01:15:00.400899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.400871 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:00.401124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.401001 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf"] Oct 30 01:15:00.405514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.405482 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686"] Oct 30 01:15:00.412069 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.412040 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6"] Oct 30 01:15:00.412256 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.412157 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:00.415419 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.415382 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686"] Oct 30 01:15:00.472778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.472736 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d051c589-6cf9-4544-896f-5cc2d09feee3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w\" (UID: \"d051c589-6cf9-4544-896f-5cc2d09feee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:00.473089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.472860 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rq9\" (UniqueName: \"kubernetes.io/projected/98219cb4-5af9-4242-8524-12cc774bd3fc-kube-api-access-j6rq9\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh\" (UID: \"98219cb4-5af9-4242-8524-12cc774bd3fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:00.473089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.472912 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dswp\" (UniqueName: \"kubernetes.io/projected/953fecbb-4d1a-4626-ac60-dd0825d89a5e-kube-api-access-2dswp\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6\" (UID: \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:00.473089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473076 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqt8t\" (UniqueName: \"kubernetes.io/projected/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-kube-api-access-kqt8t\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw\" (UID: \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:00.473300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473125 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646cf\" (UniqueName: \"kubernetes.io/projected/d051c589-6cf9-4544-896f-5cc2d09feee3-kube-api-access-646cf\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w\" (UID: \"d051c589-6cf9-4544-896f-5cc2d09feee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:00.473300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473166 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f77bcc-c251-49e5-a309-9dede3b87810-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs\" (UID: \"10f77bcc-c251-49e5-a309-9dede3b87810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:00.473300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473209 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09d6932-71e7-4876-81a8-8964658c2b98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm\" (UID: \"a09d6932-71e7-4876-81a8-8964658c2b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:00.473300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473225 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvlvb\" (UniqueName: \"kubernetes.io/projected/a09d6932-71e7-4876-81a8-8964658c2b98-kube-api-access-fvlvb\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm\" (UID: \"a09d6932-71e7-4876-81a8-8964658c2b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:00.473300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473258 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855240f-d1f1-47c1-b8e4-d155a4ee590f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw\" (UID: \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473308 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95n7l\" (UniqueName: \"kubernetes.io/projected/10f77bcc-c251-49e5-a309-9dede3b87810-kube-api-access-95n7l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs\" (UID: \"10f77bcc-c251-49e5-a309-9dede3b87810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473332 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sn9c\" (UniqueName: \"kubernetes.io/projected/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-kube-api-access-5sn9c\") pod \"maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd\" (UID: \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473358 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7826a2-7045-4db6-9952-fbc16ddc5e22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf\" (UID: \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473378 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw\" (UID: \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473412 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686\" (UID: \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473432 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5sgc\" (UniqueName: \"kubernetes.io/projected/f855240f-d1f1-47c1-b8e4-d155a4ee590f-kube-api-access-l5sgc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw\" (UID: \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473449 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tt2b\" (UniqueName: \"kubernetes.io/projected/1d7826a2-7045-4db6-9952-fbc16ddc5e22-kube-api-access-7tt2b\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf\" (UID: \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473475 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rzj\" (UniqueName: \"kubernetes.io/projected/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-kube-api-access-47rzj\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686\" (UID: \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473495 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953fecbb-4d1a-4626-ac60-dd0825d89a5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6\" (UID: \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473516 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd\" (UID: \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:00.473557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473549 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98219cb4-5af9-4242-8524-12cc774bd3fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh\" (UID: \"98219cb4-5af9-4242-8524-12cc774bd3fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:00.474250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.473755 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d051c589-6cf9-4544-896f-5cc2d09feee3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w\" (UID: \"d051c589-6cf9-4544-896f-5cc2d09feee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:00.474676 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.474633 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd\" (UID: \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:00.474988 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.474960 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f77bcc-c251-49e5-a309-9dede3b87810-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs\" (UID: \"10f77bcc-c251-49e5-a309-9dede3b87810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:00.475201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.475172 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855240f-d1f1-47c1-b8e4-d155a4ee590f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw\" (UID: \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:00.475326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.475244 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:00.475326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.475279 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09d6932-71e7-4876-81a8-8964658c2b98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm\" (UID: \"a09d6932-71e7-4876-81a8-8964658c2b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:00.496487 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.496446 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvlvb\" (UniqueName: \"kubernetes.io/projected/a09d6932-71e7-4876-81a8-8964658c2b98-kube-api-access-fvlvb\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm\" (UID: \"a09d6932-71e7-4876-81a8-8964658c2b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:00.497230 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.497199 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sn9c\" (UniqueName: \"kubernetes.io/projected/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-kube-api-access-5sn9c\") pod \"maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd\" (UID: \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:00.497740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.497699 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-646cf\" (UniqueName: \"kubernetes.io/projected/d051c589-6cf9-4544-896f-5cc2d09feee3-kube-api-access-646cf\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w\" (UID: \"d051c589-6cf9-4544-896f-5cc2d09feee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:00.498185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.498164 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5sgc\" (UniqueName: \"kubernetes.io/projected/f855240f-d1f1-47c1-b8e4-d155a4ee590f-kube-api-access-l5sgc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw\" (UID: \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:00.500142 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.500119 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95n7l\" (UniqueName: \"kubernetes.io/projected/10f77bcc-c251-49e5-a309-9dede3b87810-kube-api-access-95n7l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs\" (UID: \"10f77bcc-c251-49e5-a309-9dede3b87810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:00.527774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.527682 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:00.614826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.611059 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:00.615189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615155 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7826a2-7045-4db6-9952-fbc16ddc5e22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf\" (UID: \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:00.615309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615210 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw\" (UID: \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:00.615309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615265 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686\" (UID: \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:00.615309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615301 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tt2b\" (UniqueName: \"kubernetes.io/projected/1d7826a2-7045-4db6-9952-fbc16ddc5e22-kube-api-access-7tt2b\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf\" (UID: \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:00.615493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615333 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rzj\" (UniqueName: \"kubernetes.io/projected/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-kube-api-access-47rzj\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686\" (UID: \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:00.615493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615366 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953fecbb-4d1a-4626-ac60-dd0825d89a5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6\" (UID: \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:00.615493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615415 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98219cb4-5af9-4242-8524-12cc774bd3fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh\" (UID: \"98219cb4-5af9-4242-8524-12cc774bd3fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:00.615657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615535 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rq9\" (UniqueName: \"kubernetes.io/projected/98219cb4-5af9-4242-8524-12cc774bd3fc-kube-api-access-j6rq9\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh\" (UID: \"98219cb4-5af9-4242-8524-12cc774bd3fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:00.615657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615583 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dswp\" (UniqueName: \"kubernetes.io/projected/953fecbb-4d1a-4626-ac60-dd0825d89a5e-kube-api-access-2dswp\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6\" (UID: \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:00.615860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.615677 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqt8t\" (UniqueName: \"kubernetes.io/projected/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-kube-api-access-kqt8t\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw\" (UID: \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:00.620546 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.620509 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw\" (UID: \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:00.620702 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.620682 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98219cb4-5af9-4242-8524-12cc774bd3fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh\" (UID: \"98219cb4-5af9-4242-8524-12cc774bd3fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:00.626211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.626173 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:00.626589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.626560 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7826a2-7045-4db6-9952-fbc16ddc5e22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf\" (UID: \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:00.629591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.627475 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686\" (UID: \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:00.629591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.627697 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953fecbb-4d1a-4626-ac60-dd0825d89a5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6\" (UID: \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:00.634410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.634360 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:00.640052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.640020 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:00.643250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.643212 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rzj\" (UniqueName: \"kubernetes.io/projected/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-kube-api-access-47rzj\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686\" (UID: \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:00.657841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.655906 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:15:00.657841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.657658 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:15:00.667385 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.662896 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rq9\" (UniqueName: \"kubernetes.io/projected/98219cb4-5af9-4242-8524-12cc774bd3fc-kube-api-access-j6rq9\") pod \"maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh\" (UID: \"98219cb4-5af9-4242-8524-12cc774bd3fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:00.676161 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.675788 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tt2b\" (UniqueName: \"kubernetes.io/projected/1d7826a2-7045-4db6-9952-fbc16ddc5e22-kube-api-access-7tt2b\") pod \"maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf\" (UID: \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:00.676394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.676173 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqt8t\" (UniqueName: \"kubernetes.io/projected/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-kube-api-access-kqt8t\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw\" (UID: \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:00.679741 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.676933 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:15:00.715126 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.686926 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:00.728740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.719482 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:00.759812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.734088 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:00.814795 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:00.800660 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:01.045824 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.028486 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:01.194794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.194725 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dswp\" (UniqueName: \"kubernetes.io/projected/953fecbb-4d1a-4626-ac60-dd0825d89a5e-kube-api-access-2dswp\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6\" (UID: \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:01.232673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.224621 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:01.234000 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:01.233920 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:15:01.261155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.261107 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4"] Oct 30 01:15:01.346301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.345994 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:01.465941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.460452 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:01.606099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.590564 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm"] Oct 30 01:15:01.606099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.593654 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:01.631939 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.631884 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w"] Oct 30 01:15:01.902911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:01.902868 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:02.704680 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:02.704639 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd"] Oct 30 01:15:02.705691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:02.705284 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs"] Oct 30 01:15:02.705977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:02.705945 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh"] Oct 30 01:15:02.708190 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:02.707965 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" event={"ID":"e7837d7d-e6f2-45cc-9258-d29fdfe6764f","Type":"ContainerStarted","Data":"f8e0f0454e0cd2d6e561012f14847b694c64ff771043916301b3927eb1de0a99"} Oct 30 01:15:02.713170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:02.713133 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" event={"ID":"a09d6932-71e7-4876-81a8-8964658c2b98","Type":"ContainerStarted","Data":"987acf0120012cb8aceac3b14020cab3dcef38c85021ae4fd21e323d829ed2c2"} Oct 30 01:15:02.715279 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:02.715188 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" event={"ID":"d051c589-6cf9-4544-896f-5cc2d09feee3","Type":"ContainerStarted","Data":"16b73b68b0918acd519f5c39165ca175087585cf230024e6594c5435411d1c91"} Oct 30 01:15:02.724295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:02.722896 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" event={"ID":"f34effdd-fdf2-49f3-b971-b12b0315257b","Type":"ContainerStarted","Data":"0559cecf0d122f0fb6335a848ce06d13f848e35752fd4911e5e7b465f9ae4081"} Oct 30 01:15:02.740361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:02.740327 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:03.083817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.080466 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:03.223083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.223018 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686"] Oct 30 01:15:03.230781 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.227614 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf"] Oct 30 01:15:03.238560 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.238338 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw"] Oct 30 01:15:03.243324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.243297 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw"] Oct 30 01:15:03.286289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.286249 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6"] Oct 30 01:15:03.288153 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:15:03.286995 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdc7cb4_c3d9_4d7f_917f_e427552083f1.slice/crio-fdd19c6dfefe3cbd78b6dde7550b8be0eabbaf2d50b1a8768a2444c306b18e3e WatchSource:0}: Error finding container fdd19c6dfefe3cbd78b6dde7550b8be0eabbaf2d50b1a8768a2444c306b18e3e: Status 404 returned error can't find the container with id fdd19c6dfefe3cbd78b6dde7550b8be0eabbaf2d50b1a8768a2444c306b18e3e Oct 30 01:15:03.307216 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:15:03.306997 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7826a2_7045_4db6_9952_fbc16ddc5e22.slice/crio-61777ce3b497776d94ea1b155aa31768ebd71645cc353ac0ae2e45b2f13787f7 WatchSource:0}: Error finding container 61777ce3b497776d94ea1b155aa31768ebd71645cc353ac0ae2e45b2f13787f7: Status 404 returned error can't find the container with id 61777ce3b497776d94ea1b155aa31768ebd71645cc353ac0ae2e45b2f13787f7 Oct 30 01:15:03.733563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.733515 4517 generic.go:356] "Generic (PLEG): container finished" podID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerID="096024f22ca21a000ba6f88b06693d7c56c17a2332c1618af72fdffd6e35f461" exitCode=60 Oct 30 01:15:03.734087 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.733571 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" event={"ID":"f34effdd-fdf2-49f3-b971-b12b0315257b","Type":"ContainerDied","Data":"096024f22ca21a000ba6f88b06693d7c56c17a2332c1618af72fdffd6e35f461"} Oct 30 01:15:03.734159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.734092 4517 scope.go:122] "RemoveContainer" containerID="096024f22ca21a000ba6f88b06693d7c56c17a2332c1618af72fdffd6e35f461" Oct 30 01:15:03.735466 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.735423 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" event={"ID":"953fecbb-4d1a-4626-ac60-dd0825d89a5e","Type":"ContainerStarted","Data":"ed0f3dca4fe5e2c7a392fe2767f9f8c739ea7de3a6bcc0a6d159a9fdfe3e5b4a"} Oct 30 01:15:03.736937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.736911 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" event={"ID":"f855240f-d1f1-47c1-b8e4-d155a4ee590f","Type":"ContainerStarted","Data":"c741d57fa0b4773221c703e78055d1b7bca0b1a606ca3919918f64cb5419f536"} Oct 30 01:15:03.739059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.739015 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" event={"ID":"98219cb4-5af9-4242-8524-12cc774bd3fc","Type":"ContainerStarted","Data":"33f90cc78fd4f50d9486e1f60be1108ef8b64bc8b4029f8cf490315d735e6610"} Oct 30 01:15:03.739176 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.739087 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" event={"ID":"98219cb4-5af9-4242-8524-12cc774bd3fc","Type":"ContainerStarted","Data":"c16a7a9d0947b91a6cffe062713e0b1ca7fa0550efba38ffa0ffd6920fc0dc68"} Oct 30 01:15:03.741505 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.741460 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" event={"ID":"e7837d7d-e6f2-45cc-9258-d29fdfe6764f","Type":"ContainerStarted","Data":"cf937c6b55f149b86a535c0203a58b7e157c79957352289c7be9044c480914cf"} Oct 30 01:15:03.743750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.743701 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" event={"ID":"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8","Type":"ContainerStarted","Data":"507170f7c27f8b7f3271f38c8337cac95fe2233bee60fdc02e5088b25fdbc891"} Oct 30 01:15:03.745658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.745626 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" event={"ID":"d051c589-6cf9-4544-896f-5cc2d09feee3","Type":"ContainerStarted","Data":"656b142e7ac305c7a7454c6101fa5d99928d3f48fe391a9066c37d406f6e921d"} Oct 30 01:15:03.747598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.747563 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" event={"ID":"1d7826a2-7045-4db6-9952-fbc16ddc5e22","Type":"ContainerStarted","Data":"61777ce3b497776d94ea1b155aa31768ebd71645cc353ac0ae2e45b2f13787f7"} Oct 30 01:15:03.749253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.749221 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" event={"ID":"7bdc7cb4-c3d9-4d7f-917f-e427552083f1","Type":"ContainerStarted","Data":"fdd19c6dfefe3cbd78b6dde7550b8be0eabbaf2d50b1a8768a2444c306b18e3e"} Oct 30 01:15:03.751209 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.751183 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" event={"ID":"10f77bcc-c251-49e5-a309-9dede3b87810","Type":"ContainerStarted","Data":"6e052d7177e9557f56d9b7f02b3e8c4a1c14c03231568f6a4734be9f91b07c87"} Oct 30 01:15:03.751346 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.751235 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" event={"ID":"10f77bcc-c251-49e5-a309-9dede3b87810","Type":"ContainerStarted","Data":"47421adbea8dac55ba6f2c7dbccc435f7846c359872106f4b59605b5f6890d4e"} Oct 30 01:15:03.753391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.753361 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" event={"ID":"a09d6932-71e7-4876-81a8-8964658c2b98","Type":"ContainerStarted","Data":"749b726dd6242f69a95883579f12ffb37f714231f489b09c4b569287ff59b988"} Oct 30 01:15:03.937731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.935389 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" podStartSLOduration=3.935364686 podStartE2EDuration="3.935364686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.809609103 +0000 UTC m=+2441.721736245" watchObservedRunningTime="2026-10-30 01:15:03.935364686 +0000 UTC m=+2441.847491830" Oct 30 01:15:03.949134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.944176 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" podStartSLOduration=3.944153074 podStartE2EDuration="3.944153074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.851627523 +0000 UTC m=+2441.763754689" watchObservedRunningTime="2026-10-30 01:15:03.944153074 +0000 UTC m=+2441.856280214" Oct 30 01:15:03.950924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.950865 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" podStartSLOduration=3.950847959 podStartE2EDuration="3.950847959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.907281189 +0000 UTC m=+2441.819408358" watchObservedRunningTime="2026-10-30 01:15:03.950847959 +0000 UTC m=+2441.862975099" Oct 30 01:15:03.958594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.957935 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" podStartSLOduration=3.957917521 podStartE2EDuration="3.957917521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.945556194 +0000 UTC m=+2441.857683336" watchObservedRunningTime="2026-10-30 01:15:03.957917521 +0000 UTC m=+2441.870044661" Oct 30 01:15:03.991728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:03.991652 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" podStartSLOduration=3.9916356840000002 podStartE2EDuration="3.991635684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.985877502 +0000 UTC m=+2441.898004668" watchObservedRunningTime="2026-10-30 01:15:03.991635684 +0000 UTC m=+2441.903762825" Oct 30 01:15:04.774916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.774875 4517 generic.go:356] "Generic (PLEG): container finished" podID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerID="cf937c6b55f149b86a535c0203a58b7e157c79957352289c7be9044c480914cf" exitCode=60 Oct 30 01:15:04.778294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.776442 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" event={"ID":"e7837d7d-e6f2-45cc-9258-d29fdfe6764f","Type":"ContainerDied","Data":"cf937c6b55f149b86a535c0203a58b7e157c79957352289c7be9044c480914cf"} Oct 30 01:15:04.780533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.779217 4517 scope.go:122] "RemoveContainer" containerID="cf937c6b55f149b86a535c0203a58b7e157c79957352289c7be9044c480914cf" Oct 30 01:15:04.781916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.781884 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" event={"ID":"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8","Type":"ContainerStarted","Data":"ba7a897b49f5cd242ec8d946083e9af6bbb9183c2beaee3162fe3cbd7c5b30ce"} Oct 30 01:15:04.791699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.789440 4517 generic.go:356] "Generic (PLEG): container finished" podID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerID="656b142e7ac305c7a7454c6101fa5d99928d3f48fe391a9066c37d406f6e921d" exitCode=60 Oct 30 01:15:04.791699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.789857 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" event={"ID":"d051c589-6cf9-4544-896f-5cc2d09feee3","Type":"ContainerDied","Data":"656b142e7ac305c7a7454c6101fa5d99928d3f48fe391a9066c37d406f6e921d"} Oct 30 01:15:04.791699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.790130 4517 scope.go:122] "RemoveContainer" containerID="656b142e7ac305c7a7454c6101fa5d99928d3f48fe391a9066c37d406f6e921d" Oct 30 01:15:04.792289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.792251 4517 generic.go:356] "Generic (PLEG): container finished" podID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerID="eab8721872cbed24c1b492b37556426d7c6f5ef6eb233617e3f23aef6bfbc5cc" exitCode=60 Oct 30 01:15:04.792433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.792385 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" event={"ID":"1d7826a2-7045-4db6-9952-fbc16ddc5e22","Type":"ContainerDied","Data":"eab8721872cbed24c1b492b37556426d7c6f5ef6eb233617e3f23aef6bfbc5cc"} Oct 30 01:15:04.792943 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.792919 4517 scope.go:122] "RemoveContainer" containerID="eab8721872cbed24c1b492b37556426d7c6f5ef6eb233617e3f23aef6bfbc5cc" Oct 30 01:15:04.796152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.796114 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" event={"ID":"7bdc7cb4-c3d9-4d7f-917f-e427552083f1","Type":"ContainerStarted","Data":"9c276edde1fdaa8a21e2b64c003034e23bb40d288d9ad721aa65d1d4b41b3563"} Oct 30 01:15:04.799958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.799919 4517 generic.go:356] "Generic (PLEG): container finished" podID="10f77bcc-c251-49e5-a309-9dede3b87810" containerID="6e052d7177e9557f56d9b7f02b3e8c4a1c14c03231568f6a4734be9f91b07c87" exitCode=60 Oct 30 01:15:04.800102 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.800057 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" event={"ID":"10f77bcc-c251-49e5-a309-9dede3b87810","Type":"ContainerDied","Data":"6e052d7177e9557f56d9b7f02b3e8c4a1c14c03231568f6a4734be9f91b07c87"} Oct 30 01:15:04.800594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.800567 4517 scope.go:122] "RemoveContainer" containerID="6e052d7177e9557f56d9b7f02b3e8c4a1c14c03231568f6a4734be9f91b07c87" Oct 30 01:15:04.837678 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.835304 4517 generic.go:356] "Generic (PLEG): container finished" podID="a09d6932-71e7-4876-81a8-8964658c2b98" containerID="749b726dd6242f69a95883579f12ffb37f714231f489b09c4b569287ff59b988" exitCode=60 Oct 30 01:15:04.837678 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.835418 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" event={"ID":"a09d6932-71e7-4876-81a8-8964658c2b98","Type":"ContainerDied","Data":"749b726dd6242f69a95883579f12ffb37f714231f489b09c4b569287ff59b988"} Oct 30 01:15:04.837678 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.835963 4517 scope.go:122] "RemoveContainer" containerID="749b726dd6242f69a95883579f12ffb37f714231f489b09c4b569287ff59b988" Oct 30 01:15:04.845364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.844347 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" event={"ID":"f34effdd-fdf2-49f3-b971-b12b0315257b","Type":"ContainerStarted","Data":"b639948aebca2aaff4f98c398d2af983b53dba1f262bafd056abee4acc3c5e3d"} Oct 30 01:15:04.855409 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.855042 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" event={"ID":"953fecbb-4d1a-4626-ac60-dd0825d89a5e","Type":"ContainerStarted","Data":"e967b79507ad1f5d7108bccf060ad84b4ec748d12e50af4f6df40063225f9c0e"} Oct 30 01:15:04.855569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.855544 4517 scope.go:122] "RemoveContainer" containerID="e967b79507ad1f5d7108bccf060ad84b4ec748d12e50af4f6df40063225f9c0e" Oct 30 01:15:04.859928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.858702 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" event={"ID":"f855240f-d1f1-47c1-b8e4-d155a4ee590f","Type":"ContainerStarted","Data":"9d0d5e3e2616870c19ab25f769acfd40900336f8fa1910a206aa6942e6fc4d73"} Oct 30 01:15:04.859928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.859180 4517 scope.go:122] "RemoveContainer" containerID="9d0d5e3e2616870c19ab25f769acfd40900336f8fa1910a206aa6942e6fc4d73" Oct 30 01:15:04.862758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.862657 4517 generic.go:356] "Generic (PLEG): container finished" podID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerID="33f90cc78fd4f50d9486e1f60be1108ef8b64bc8b4029f8cf490315d735e6610" exitCode=60 Oct 30 01:15:04.862916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.862898 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" event={"ID":"98219cb4-5af9-4242-8524-12cc774bd3fc","Type":"ContainerDied","Data":"33f90cc78fd4f50d9486e1f60be1108ef8b64bc8b4029f8cf490315d735e6610"} Oct 30 01:15:04.863677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.863325 4517 scope.go:122] "RemoveContainer" containerID="33f90cc78fd4f50d9486e1f60be1108ef8b64bc8b4029f8cf490315d735e6610" Oct 30 01:15:04.940329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.940262 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" podStartSLOduration=4.940247759 podStartE2EDuration="4.940247759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.939591209 +0000 UTC m=+2442.851718350" watchObservedRunningTime="2026-10-30 01:15:04.940247759 +0000 UTC m=+2442.852374900" Oct 30 01:15:04.983458 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:04.983385 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" podStartSLOduration=4.983365564 podStartE2EDuration="4.983365564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.97761585 +0000 UTC m=+2442.889742992" watchObservedRunningTime="2026-10-30 01:15:04.983365564 +0000 UTC m=+2442.895492701" Oct 30 01:15:05.063372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.063302 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" podStartSLOduration=5.063276699 podStartE2EDuration="5.063276699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:05.056122281 +0000 UTC m=+2442.968249424" watchObservedRunningTime="2026-10-30 01:15:05.063276699 +0000 UTC m=+2442.975403843" Oct 30 01:15:05.245171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.245134 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:05.875698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.875657 4517 generic.go:356] "Generic (PLEG): container finished" podID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerID="9d0d5e3e2616870c19ab25f769acfd40900336f8fa1910a206aa6942e6fc4d73" exitCode=60 Oct 30 01:15:05.876210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.875774 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" event={"ID":"f855240f-d1f1-47c1-b8e4-d155a4ee590f","Type":"ContainerDied","Data":"9d0d5e3e2616870c19ab25f769acfd40900336f8fa1910a206aa6942e6fc4d73"} Oct 30 01:15:05.876210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.875874 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" event={"ID":"f855240f-d1f1-47c1-b8e4-d155a4ee590f","Type":"ContainerStarted","Data":"bd6f8a2c9a8bfcb621532d1d6aaba5a35b39af9b135e1a7015f3087402693fdc"} Oct 30 01:15:05.878574 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.878541 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" event={"ID":"98219cb4-5af9-4242-8524-12cc774bd3fc","Type":"ContainerStarted","Data":"c7131ce93112f7460913c53ba353e44005bb94ae3594d5ce09170a81009d3a22"} Oct 30 01:15:05.881593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.880910 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" event={"ID":"e7837d7d-e6f2-45cc-9258-d29fdfe6764f","Type":"ContainerStarted","Data":"c2540c3f2bb38126c503ff235c0b4a9a93ecde372429f013f13227f473f56bf6"} Oct 30 01:15:05.883001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.882972 4517 generic.go:356] "Generic (PLEG): container finished" podID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerID="ba7a897b49f5cd242ec8d946083e9af6bbb9183c2beaee3162fe3cbd7c5b30ce" exitCode=60 Oct 30 01:15:05.883180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.883058 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" event={"ID":"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8","Type":"ContainerDied","Data":"ba7a897b49f5cd242ec8d946083e9af6bbb9183c2beaee3162fe3cbd7c5b30ce"} Oct 30 01:15:05.883360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.883340 4517 scope.go:122] "RemoveContainer" containerID="ba7a897b49f5cd242ec8d946083e9af6bbb9183c2beaee3162fe3cbd7c5b30ce" Oct 30 01:15:05.887330 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.887303 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" event={"ID":"d051c589-6cf9-4544-896f-5cc2d09feee3","Type":"ContainerStarted","Data":"12f58e56ca711819f368e1d54a9c212ca7a4fd141f71e550cea20f48b83857f2"} Oct 30 01:15:05.891185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.891154 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" event={"ID":"1d7826a2-7045-4db6-9952-fbc16ddc5e22","Type":"ContainerStarted","Data":"5d81bb8c2352479594d39197f843f786bcaaf5c81e6c10a93d8624542db9ed1e"} Oct 30 01:15:05.895185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.894949 4517 generic.go:356] "Generic (PLEG): container finished" podID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerID="9c276edde1fdaa8a21e2b64c003034e23bb40d288d9ad721aa65d1d4b41b3563" exitCode=60 Oct 30 01:15:05.895185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.895057 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" event={"ID":"7bdc7cb4-c3d9-4d7f-917f-e427552083f1","Type":"ContainerDied","Data":"9c276edde1fdaa8a21e2b64c003034e23bb40d288d9ad721aa65d1d4b41b3563"} Oct 30 01:15:05.895558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.895537 4517 scope.go:122] "RemoveContainer" containerID="9c276edde1fdaa8a21e2b64c003034e23bb40d288d9ad721aa65d1d4b41b3563" Oct 30 01:15:05.899236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.899168 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" event={"ID":"10f77bcc-c251-49e5-a309-9dede3b87810","Type":"ContainerStarted","Data":"36c75a27c02292d15c2c0b26149eac211403e90179cf67ec84c014b4a51e1db2"} Oct 30 01:15:05.903669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.903639 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" event={"ID":"a09d6932-71e7-4876-81a8-8964658c2b98","Type":"ContainerStarted","Data":"df3a0af876ca00afa07b439131e6bcab8952632d613c9140845fa6ed3f7d61e3"} Oct 30 01:15:05.906956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.906924 4517 generic.go:356] "Generic (PLEG): container finished" podID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerID="b639948aebca2aaff4f98c398d2af983b53dba1f262bafd056abee4acc3c5e3d" exitCode=60 Oct 30 01:15:05.907087 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.907007 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" event={"ID":"f34effdd-fdf2-49f3-b971-b12b0315257b","Type":"ContainerDied","Data":"b639948aebca2aaff4f98c398d2af983b53dba1f262bafd056abee4acc3c5e3d"} Oct 30 01:15:05.907468 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.907433 4517 scope.go:122] "RemoveContainer" containerID="b639948aebca2aaff4f98c398d2af983b53dba1f262bafd056abee4acc3c5e3d" Oct 30 01:15:05.907560 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.907515 4517 scope.go:122] "RemoveContainer" containerID="096024f22ca21a000ba6f88b06693d7c56c17a2332c1618af72fdffd6e35f461" Oct 30 01:15:05.907882 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:05.907856 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4_odh-ai-gateway-infra(f34effdd-fdf2-49f3-b971-b12b0315257b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" Oct 30 01:15:05.909897 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.909870 4517 generic.go:356] "Generic (PLEG): container finished" podID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerID="e967b79507ad1f5d7108bccf060ad84b4ec748d12e50af4f6df40063225f9c0e" exitCode=60 Oct 30 01:15:05.910012 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.909941 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" event={"ID":"953fecbb-4d1a-4626-ac60-dd0825d89a5e","Type":"ContainerDied","Data":"e967b79507ad1f5d7108bccf060ad84b4ec748d12e50af4f6df40063225f9c0e"} Oct 30 01:15:05.910012 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.909988 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" event={"ID":"953fecbb-4d1a-4626-ac60-dd0825d89a5e","Type":"ContainerStarted","Data":"48eddee4e6d086b6f483e9cf12166c5e85afeda3b321948c0f7ebde990b8f362"} Oct 30 01:15:05.914563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:05.914502 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" podStartSLOduration=5.914484526 podStartE2EDuration="5.914484526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:05.911861557 +0000 UTC m=+2443.823988698" watchObservedRunningTime="2026-10-30 01:15:05.914484526 +0000 UTC m=+2443.826611669" Oct 30 01:15:06.112852 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.112779 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" podStartSLOduration=6.112761536 podStartE2EDuration="6.112761536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.106585884 +0000 UTC m=+2444.018713027" watchObservedRunningTime="2026-10-30 01:15:06.112761536 +0000 UTC m=+2444.024888677" Oct 30 01:15:06.148181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.148062 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" podStartSLOduration=6.14804003 podStartE2EDuration="6.14804003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.143033969 +0000 UTC m=+2444.055161089" watchObservedRunningTime="2026-10-30 01:15:06.14804003 +0000 UTC m=+2444.060167176" Oct 30 01:15:06.217597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.217374 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:06.347823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.347782 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:06.927683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.927643 4517 generic.go:356] "Generic (PLEG): container finished" podID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerID="48eddee4e6d086b6f483e9cf12166c5e85afeda3b321948c0f7ebde990b8f362" exitCode=60 Oct 30 01:15:06.928141 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.927750 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" event={"ID":"953fecbb-4d1a-4626-ac60-dd0825d89a5e","Type":"ContainerDied","Data":"48eddee4e6d086b6f483e9cf12166c5e85afeda3b321948c0f7ebde990b8f362"} Oct 30 01:15:06.928141 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.927820 4517 scope.go:122] "RemoveContainer" containerID="e967b79507ad1f5d7108bccf060ad84b4ec748d12e50af4f6df40063225f9c0e" Oct 30 01:15:06.928297 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.928218 4517 scope.go:122] "RemoveContainer" containerID="48eddee4e6d086b6f483e9cf12166c5e85afeda3b321948c0f7ebde990b8f362" Oct 30 01:15:06.928698 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:06.928668 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6_odh-ai-gateway-infra(953fecbb-4d1a-4626-ac60-dd0825d89a5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" Oct 30 01:15:06.930492 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.930469 4517 generic.go:356] "Generic (PLEG): container finished" podID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerID="bd6f8a2c9a8bfcb621532d1d6aaba5a35b39af9b135e1a7015f3087402693fdc" exitCode=60 Oct 30 01:15:06.930591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.930547 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" event={"ID":"f855240f-d1f1-47c1-b8e4-d155a4ee590f","Type":"ContainerDied","Data":"bd6f8a2c9a8bfcb621532d1d6aaba5a35b39af9b135e1a7015f3087402693fdc"} Oct 30 01:15:06.930881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.930864 4517 scope.go:122] "RemoveContainer" containerID="bd6f8a2c9a8bfcb621532d1d6aaba5a35b39af9b135e1a7015f3087402693fdc" Oct 30 01:15:06.931293 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:06.931265 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw_odh-ai-gateway-infra(f855240f-d1f1-47c1-b8e4-d155a4ee590f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" Oct 30 01:15:06.932753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.932729 4517 generic.go:356] "Generic (PLEG): container finished" podID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerID="c7131ce93112f7460913c53ba353e44005bb94ae3594d5ce09170a81009d3a22" exitCode=60 Oct 30 01:15:06.932840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.932813 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" event={"ID":"98219cb4-5af9-4242-8524-12cc774bd3fc","Type":"ContainerDied","Data":"c7131ce93112f7460913c53ba353e44005bb94ae3594d5ce09170a81009d3a22"} Oct 30 01:15:06.933159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.933141 4517 scope.go:122] "RemoveContainer" containerID="c7131ce93112f7460913c53ba353e44005bb94ae3594d5ce09170a81009d3a22" Oct 30 01:15:06.933563 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:06.933540 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh_odh-ai-gateway-infra(98219cb4-5af9-4242-8524-12cc774bd3fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" Oct 30 01:15:06.935084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.935063 4517 generic.go:356] "Generic (PLEG): container finished" podID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerID="c2540c3f2bb38126c503ff235c0b4a9a93ecde372429f013f13227f473f56bf6" exitCode=60 Oct 30 01:15:06.935172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.935137 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" event={"ID":"e7837d7d-e6f2-45cc-9258-d29fdfe6764f","Type":"ContainerDied","Data":"c2540c3f2bb38126c503ff235c0b4a9a93ecde372429f013f13227f473f56bf6"} Oct 30 01:15:06.935509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.935492 4517 scope.go:122] "RemoveContainer" containerID="c2540c3f2bb38126c503ff235c0b4a9a93ecde372429f013f13227f473f56bf6" Oct 30 01:15:06.935883 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:06.935862 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd_odh-ai-gateway-infra(e7837d7d-e6f2-45cc-9258-d29fdfe6764f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" Oct 30 01:15:06.937336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.937315 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" event={"ID":"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8","Type":"ContainerStarted","Data":"b792999e89f7c0e9b372faa952beaa6aca1d01c2218ec86bbd5bdf0ef0bc39c3"} Oct 30 01:15:06.939472 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.939449 4517 generic.go:356] "Generic (PLEG): container finished" podID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerID="12f58e56ca711819f368e1d54a9c212ca7a4fd141f71e550cea20f48b83857f2" exitCode=60 Oct 30 01:15:06.939587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.939535 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" event={"ID":"d051c589-6cf9-4544-896f-5cc2d09feee3","Type":"ContainerDied","Data":"12f58e56ca711819f368e1d54a9c212ca7a4fd141f71e550cea20f48b83857f2"} Oct 30 01:15:06.939870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.939851 4517 scope.go:122] "RemoveContainer" containerID="12f58e56ca711819f368e1d54a9c212ca7a4fd141f71e550cea20f48b83857f2" Oct 30 01:15:06.940255 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:06.940233 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w_odh-ai-gateway-infra(d051c589-6cf9-4544-896f-5cc2d09feee3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" Oct 30 01:15:06.941621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.941598 4517 generic.go:356] "Generic (PLEG): container finished" podID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerID="5d81bb8c2352479594d39197f843f786bcaaf5c81e6c10a93d8624542db9ed1e" exitCode=60 Oct 30 01:15:06.941762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.941697 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" event={"ID":"1d7826a2-7045-4db6-9952-fbc16ddc5e22","Type":"ContainerDied","Data":"5d81bb8c2352479594d39197f843f786bcaaf5c81e6c10a93d8624542db9ed1e"} Oct 30 01:15:06.941921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.941906 4517 scope.go:122] "RemoveContainer" containerID="5d81bb8c2352479594d39197f843f786bcaaf5c81e6c10a93d8624542db9ed1e" Oct 30 01:15:06.942234 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:06.942213 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf_odh-ai-gateway-infra(1d7826a2-7045-4db6-9952-fbc16ddc5e22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" Oct 30 01:15:06.943992 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.943848 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" event={"ID":"7bdc7cb4-c3d9-4d7f-917f-e427552083f1","Type":"ContainerStarted","Data":"58ba8fb4044296f84e4c163313d79974d885cee9dd5d6b766407f5f2c665faa0"} Oct 30 01:15:06.944058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.944007 4517 scope.go:122] "RemoveContainer" containerID="9d0d5e3e2616870c19ab25f769acfd40900336f8fa1910a206aa6942e6fc4d73" Oct 30 01:15:06.946021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.946001 4517 generic.go:356] "Generic (PLEG): container finished" podID="10f77bcc-c251-49e5-a309-9dede3b87810" containerID="36c75a27c02292d15c2c0b26149eac211403e90179cf67ec84c014b4a51e1db2" exitCode=60 Oct 30 01:15:06.946122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.946075 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" event={"ID":"10f77bcc-c251-49e5-a309-9dede3b87810","Type":"ContainerDied","Data":"36c75a27c02292d15c2c0b26149eac211403e90179cf67ec84c014b4a51e1db2"} Oct 30 01:15:06.946306 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.946292 4517 scope.go:122] "RemoveContainer" containerID="36c75a27c02292d15c2c0b26149eac211403e90179cf67ec84c014b4a51e1db2" Oct 30 01:15:06.946597 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:06.946573 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs_odh-ai-gateway-infra(10f77bcc-c251-49e5-a309-9dede3b87810)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" Oct 30 01:15:06.947943 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.947918 4517 generic.go:356] "Generic (PLEG): container finished" podID="a09d6932-71e7-4876-81a8-8964658c2b98" containerID="df3a0af876ca00afa07b439131e6bcab8952632d613c9140845fa6ed3f7d61e3" exitCode=60 Oct 30 01:15:06.948038 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.947949 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" event={"ID":"a09d6932-71e7-4876-81a8-8964658c2b98","Type":"ContainerDied","Data":"df3a0af876ca00afa07b439131e6bcab8952632d613c9140845fa6ed3f7d61e3"} Oct 30 01:15:06.948274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.948259 4517 scope.go:122] "RemoveContainer" containerID="df3a0af876ca00afa07b439131e6bcab8952632d613c9140845fa6ed3f7d61e3" Oct 30 01:15:06.948583 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:06.948561 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm_odh-ai-gateway-infra(a09d6932-71e7-4876-81a8-8964658c2b98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" Oct 30 01:15:06.949955 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.949934 4517 scope.go:122] "RemoveContainer" containerID="b639948aebca2aaff4f98c398d2af983b53dba1f262bafd056abee4acc3c5e3d" Oct 30 01:15:06.950229 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:06.950211 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4_odh-ai-gateway-infra(f34effdd-fdf2-49f3-b971-b12b0315257b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" Oct 30 01:15:06.960084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.960051 4517 scope.go:122] "RemoveContainer" containerID="33f90cc78fd4f50d9486e1f60be1108ef8b64bc8b4029f8cf490315d735e6610" Oct 30 01:15:06.981736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:06.981690 4517 scope.go:122] "RemoveContainer" containerID="cf937c6b55f149b86a535c0203a58b7e157c79957352289c7be9044c480914cf" Oct 30 01:15:07.000326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.000296 4517 scope.go:122] "RemoveContainer" containerID="656b142e7ac305c7a7454c6101fa5d99928d3f48fe391a9066c37d406f6e921d" Oct 30 01:15:07.017335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.017305 4517 scope.go:122] "RemoveContainer" containerID="eab8721872cbed24c1b492b37556426d7c6f5ef6eb233617e3f23aef6bfbc5cc" Oct 30 01:15:07.032915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.032889 4517 scope.go:122] "RemoveContainer" containerID="6e052d7177e9557f56d9b7f02b3e8c4a1c14c03231568f6a4734be9f91b07c87" Oct 30 01:15:07.049502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.049477 4517 scope.go:122] "RemoveContainer" containerID="749b726dd6242f69a95883579f12ffb37f714231f489b09c4b569287ff59b988" Oct 30 01:15:07.215271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.215174 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:07.923278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.923230 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:07.965851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.965818 4517 scope.go:122] "RemoveContainer" containerID="bd6f8a2c9a8bfcb621532d1d6aaba5a35b39af9b135e1a7015f3087402693fdc" Oct 30 01:15:07.966368 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.966242 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw_odh-ai-gateway-infra(f855240f-d1f1-47c1-b8e4-d155a4ee590f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" Oct 30 01:15:07.967639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.967614 4517 scope.go:122] "RemoveContainer" containerID="c7131ce93112f7460913c53ba353e44005bb94ae3594d5ce09170a81009d3a22" Oct 30 01:15:07.968154 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.968124 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh_odh-ai-gateway-infra(98219cb4-5af9-4242-8524-12cc774bd3fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" Oct 30 01:15:07.970007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.969979 4517 scope.go:122] "RemoveContainer" containerID="c2540c3f2bb38126c503ff235c0b4a9a93ecde372429f013f13227f473f56bf6" Oct 30 01:15:07.970477 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.970448 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd_odh-ai-gateway-infra(e7837d7d-e6f2-45cc-9258-d29fdfe6764f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" Oct 30 01:15:07.971764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.971735 4517 generic.go:356] "Generic (PLEG): container finished" podID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerID="b792999e89f7c0e9b372faa952beaa6aca1d01c2218ec86bbd5bdf0ef0bc39c3" exitCode=60 Oct 30 01:15:07.971764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.971809 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" event={"ID":"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8","Type":"ContainerDied","Data":"b792999e89f7c0e9b372faa952beaa6aca1d01c2218ec86bbd5bdf0ef0bc39c3"} Oct 30 01:15:07.972043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.971908 4517 scope.go:122] "RemoveContainer" containerID="ba7a897b49f5cd242ec8d946083e9af6bbb9183c2beaee3162fe3cbd7c5b30ce" Oct 30 01:15:07.972516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.972347 4517 scope.go:122] "RemoveContainer" containerID="b792999e89f7c0e9b372faa952beaa6aca1d01c2218ec86bbd5bdf0ef0bc39c3" Oct 30 01:15:07.972892 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.972865 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686_odh-ai-gateway-infra(fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" Oct 30 01:15:07.974427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.974408 4517 scope.go:122] "RemoveContainer" containerID="12f58e56ca711819f368e1d54a9c212ca7a4fd141f71e550cea20f48b83857f2" Oct 30 01:15:07.974894 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.974863 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w_odh-ai-gateway-infra(d051c589-6cf9-4544-896f-5cc2d09feee3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" Oct 30 01:15:07.976542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.976522 4517 scope.go:122] "RemoveContainer" containerID="5d81bb8c2352479594d39197f843f786bcaaf5c81e6c10a93d8624542db9ed1e" Oct 30 01:15:07.976973 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.976947 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf_odh-ai-gateway-infra(1d7826a2-7045-4db6-9952-fbc16ddc5e22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" Oct 30 01:15:07.978949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.978924 4517 generic.go:356] "Generic (PLEG): container finished" podID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerID="58ba8fb4044296f84e4c163313d79974d885cee9dd5d6b766407f5f2c665faa0" exitCode=60 Oct 30 01:15:07.979075 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.978993 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" event={"ID":"7bdc7cb4-c3d9-4d7f-917f-e427552083f1","Type":"ContainerDied","Data":"58ba8fb4044296f84e4c163313d79974d885cee9dd5d6b766407f5f2c665faa0"} Oct 30 01:15:07.979462 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.979439 4517 scope.go:122] "RemoveContainer" containerID="58ba8fb4044296f84e4c163313d79974d885cee9dd5d6b766407f5f2c665faa0" Oct 30 01:15:07.980001 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.979953 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw_odh-ai-gateway-infra(7bdc7cb4-c3d9-4d7f-917f-e427552083f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" Oct 30 01:15:07.981906 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.981886 4517 scope.go:122] "RemoveContainer" containerID="36c75a27c02292d15c2c0b26149eac211403e90179cf67ec84c014b4a51e1db2" Oct 30 01:15:07.982322 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.982295 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs_odh-ai-gateway-infra(10f77bcc-c251-49e5-a309-9dede3b87810)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" Oct 30 01:15:07.984292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.984274 4517 scope.go:122] "RemoveContainer" containerID="df3a0af876ca00afa07b439131e6bcab8952632d613c9140845fa6ed3f7d61e3" Oct 30 01:15:07.984695 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.984671 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm_odh-ai-gateway-infra(a09d6932-71e7-4876-81a8-8964658c2b98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" Oct 30 01:15:07.986627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.986600 4517 scope.go:122] "RemoveContainer" containerID="48eddee4e6d086b6f483e9cf12166c5e85afeda3b321948c0f7ebde990b8f362" Oct 30 01:15:07.987363 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:07.987336 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6_odh-ai-gateway-infra(953fecbb-4d1a-4626-ac60-dd0825d89a5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" Oct 30 01:15:07.992592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:07.992561 4517 scope.go:122] "RemoveContainer" containerID="9c276edde1fdaa8a21e2b64c003034e23bb40d288d9ad721aa65d1d4b41b3563" Oct 30 01:15:08.058227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.058120 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:08.098585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.098527 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5"] Oct 30 01:15:08.106667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.106633 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:08.125813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.125771 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5"] Oct 30 01:15:08.127598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.127570 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d1ae36-dockercfg-hj2mj\"" Oct 30 01:15:08.159740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.159664 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d1ae36\"" Oct 30 01:15:08.255809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.250089 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6bf45e-8681-41e5-8d27-ebe94785eee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5\" (UID: \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:08.256499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.256458 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8df\" (UniqueName: \"kubernetes.io/projected/ef6bf45e-8681-41e5-8d27-ebe94785eee7-kube-api-access-5k8df\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5\" (UID: \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:08.365775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.359491 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8df\" (UniqueName: \"kubernetes.io/projected/ef6bf45e-8681-41e5-8d27-ebe94785eee7-kube-api-access-5k8df\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5\" (UID: \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:08.365775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.359765 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6bf45e-8681-41e5-8d27-ebe94785eee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5\" (UID: \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:08.365775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.360860 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6bf45e-8681-41e5-8d27-ebe94785eee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5\" (UID: \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:08.457742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.447488 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8df\" (UniqueName: \"kubernetes.io/projected/ef6bf45e-8681-41e5-8d27-ebe94785eee7-kube-api-access-5k8df\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5\" (UID: \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:08.729893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:08.729857 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:09.001380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:09.001250 4517 scope.go:122] "RemoveContainer" containerID="58ba8fb4044296f84e4c163313d79974d885cee9dd5d6b766407f5f2c665faa0" Oct 30 01:15:09.002673 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:09.002609 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw_odh-ai-gateway-infra(7bdc7cb4-c3d9-4d7f-917f-e427552083f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" Oct 30 01:15:09.005395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:09.005364 4517 scope.go:122] "RemoveContainer" containerID="b792999e89f7c0e9b372faa952beaa6aca1d01c2218ec86bbd5bdf0ef0bc39c3" Oct 30 01:15:09.005958 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:09.005898 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686_odh-ai-gateway-infra(fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" Oct 30 01:15:09.060481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:09.060442 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5"] Oct 30 01:15:09.062205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:09.062171 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:09.063090 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:15:09.063064 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6bf45e_8681_41e5_8d27_ebe94785eee7.slice/crio-970ab26d35b68ca3f3784ccfd8f14ab86968d1a4f360a53a7d545f07735f8be1 WatchSource:0}: Error finding container 970ab26d35b68ca3f3784ccfd8f14ab86968d1a4f360a53a7d545f07735f8be1: Status 404 returned error can't find the container with id 970ab26d35b68ca3f3784ccfd8f14ab86968d1a4f360a53a7d545f07735f8be1 Oct 30 01:15:10.039301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:10.039259 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" event={"ID":"ef6bf45e-8681-41e5-8d27-ebe94785eee7","Type":"ContainerStarted","Data":"4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a"} Oct 30 01:15:10.039850 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:10.039343 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" event={"ID":"ef6bf45e-8681-41e5-8d27-ebe94785eee7","Type":"ContainerStarted","Data":"970ab26d35b68ca3f3784ccfd8f14ab86968d1a4f360a53a7d545f07735f8be1"} Oct 30 01:15:10.040329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:10.040309 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:10.082571 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:10.082501 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" podStartSLOduration=2.082483583 podStartE2EDuration="2.082483583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:10.076750962 +0000 UTC m=+2447.988878128" watchObservedRunningTime="2026-10-30 01:15:10.082483583 +0000 UTC m=+2447.994610789" Oct 30 01:15:10.311830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:10.311728 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:15:10.312372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:10.312351 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:15:11.201495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:11.201453 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:12.993435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:12.993395 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:14.623267 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:14.623211 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:16.227899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:16.227866 4517 scope.go:122] "RemoveContainer" containerID="b639948aebca2aaff4f98c398d2af983b53dba1f262bafd056abee4acc3c5e3d" Oct 30 01:15:16.337140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:16.337090 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:16.453026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:16.452982 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:16.576650 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:16.574681 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:16.727734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:16.723327 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:17.184774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:17.184736 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" event={"ID":"f34effdd-fdf2-49f3-b971-b12b0315257b","Type":"ContainerStarted","Data":"cb453543115e4fc8cc193e83557154c8ac33861d049eed07636b0089c7900257"} Oct 30 01:15:17.244380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:17.244050 4517 scope.go:122] "RemoveContainer" containerID="df3a0af876ca00afa07b439131e6bcab8952632d613c9140845fa6ed3f7d61e3" Oct 30 01:15:17.244380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:17.244338 4517 scope.go:122] "RemoveContainer" containerID="bd6f8a2c9a8bfcb621532d1d6aaba5a35b39af9b135e1a7015f3087402693fdc" Oct 30 01:15:17.245076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:17.245045 4517 scope.go:122] "RemoveContainer" containerID="36c75a27c02292d15c2c0b26149eac211403e90179cf67ec84c014b4a51e1db2" Oct 30 01:15:17.563102 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:17.562981 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:18.211565 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.211462 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" event={"ID":"10f77bcc-c251-49e5-a309-9dede3b87810","Type":"ContainerStarted","Data":"e96d8e1daf9cbe88e342fd81009f26ca42c1b03d7a8c20b55ed87d2f0ce31e47"} Oct 30 01:15:18.214085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.214045 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" event={"ID":"a09d6932-71e7-4876-81a8-8964658c2b98","Type":"ContainerStarted","Data":"99fafa040cb40f4ca45c9e933b9c42ca6104882ca8c57d993c37424f4210806a"} Oct 30 01:15:18.216485 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.216449 4517 generic.go:356] "Generic (PLEG): container finished" podID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerID="cb453543115e4fc8cc193e83557154c8ac33861d049eed07636b0089c7900257" exitCode=60 Oct 30 01:15:18.216633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.216521 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" event={"ID":"f34effdd-fdf2-49f3-b971-b12b0315257b","Type":"ContainerDied","Data":"cb453543115e4fc8cc193e83557154c8ac33861d049eed07636b0089c7900257"} Oct 30 01:15:18.216633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.216582 4517 scope.go:122] "RemoveContainer" containerID="b639948aebca2aaff4f98c398d2af983b53dba1f262bafd056abee4acc3c5e3d" Oct 30 01:15:18.216997 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.216978 4517 scope.go:122] "RemoveContainer" containerID="cb453543115e4fc8cc193e83557154c8ac33861d049eed07636b0089c7900257" Oct 30 01:15:18.217486 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:18.217453 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4_odh-ai-gateway-infra(f34effdd-fdf2-49f3-b971-b12b0315257b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" Oct 30 01:15:18.219102 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.219073 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" event={"ID":"f855240f-d1f1-47c1-b8e4-d155a4ee590f","Type":"ContainerStarted","Data":"267778e97920af6a5b0dfed142c5fecd62d1d62f026f10cfd913ca523e29ab1f"} Oct 30 01:15:18.228463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.228430 4517 scope.go:122] "RemoveContainer" containerID="5d81bb8c2352479594d39197f843f786bcaaf5c81e6c10a93d8624542db9ed1e" Oct 30 01:15:18.229003 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.228972 4517 scope.go:122] "RemoveContainer" containerID="12f58e56ca711819f368e1d54a9c212ca7a4fd141f71e550cea20f48b83857f2" Oct 30 01:15:18.229326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.229124 4517 scope.go:122] "RemoveContainer" containerID="c7131ce93112f7460913c53ba353e44005bb94ae3594d5ce09170a81009d3a22" Oct 30 01:15:18.246231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.246194 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4"] Oct 30 01:15:18.348756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:18.348725 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:19.256738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.246241 4517 scope.go:122] "RemoveContainer" containerID="58ba8fb4044296f84e4c163313d79974d885cee9dd5d6b766407f5f2c665faa0" Oct 30 01:15:19.256738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.247343 4517 scope.go:122] "RemoveContainer" containerID="48eddee4e6d086b6f483e9cf12166c5e85afeda3b321948c0f7ebde990b8f362" Oct 30 01:15:19.362738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.351464 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs"] Oct 30 01:15:19.613752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.611126 4517 generic.go:356] "Generic (PLEG): container finished" podID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerID="267778e97920af6a5b0dfed142c5fecd62d1d62f026f10cfd913ca523e29ab1f" exitCode=60 Oct 30 01:15:19.613752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.611300 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" event={"ID":"f855240f-d1f1-47c1-b8e4-d155a4ee590f","Type":"ContainerDied","Data":"267778e97920af6a5b0dfed142c5fecd62d1d62f026f10cfd913ca523e29ab1f"} Oct 30 01:15:19.613752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.611365 4517 scope.go:122] "RemoveContainer" containerID="bd6f8a2c9a8bfcb621532d1d6aaba5a35b39af9b135e1a7015f3087402693fdc" Oct 30 01:15:19.691760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.672025 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" event={"ID":"98219cb4-5af9-4242-8524-12cc774bd3fc","Type":"ContainerStarted","Data":"351aca377c894874e967f5a4f559e00dea17e9bf800093f998911f76a69619b1"} Oct 30 01:15:19.750125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.750076 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" event={"ID":"d051c589-6cf9-4544-896f-5cc2d09feee3","Type":"ContainerStarted","Data":"68ede6897bdac3d3c28f466cefc2bda57961db6711a785be23b5ce249e5bbd28"} Oct 30 01:15:19.756744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.752407 4517 scope.go:122] "RemoveContainer" containerID="267778e97920af6a5b0dfed142c5fecd62d1d62f026f10cfd913ca523e29ab1f" Oct 30 01:15:19.756744 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:19.753140 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw_odh-ai-gateway-infra(f855240f-d1f1-47c1-b8e4-d155a4ee590f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" Oct 30 01:15:19.790739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.787268 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" event={"ID":"1d7826a2-7045-4db6-9952-fbc16ddc5e22","Type":"ContainerStarted","Data":"1ccc1046bcebdc597cda7824f9c154691558ab822dd60340546c228b941a0156"} Oct 30 01:15:19.842911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.842801 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" event={"ID":"10f77bcc-c251-49e5-a309-9dede3b87810","Type":"ContainerDied","Data":"e96d8e1daf9cbe88e342fd81009f26ca42c1b03d7a8c20b55ed87d2f0ce31e47"} Oct 30 01:15:19.851735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.824695 4517 generic.go:356] "Generic (PLEG): container finished" podID="10f77bcc-c251-49e5-a309-9dede3b87810" containerID="e96d8e1daf9cbe88e342fd81009f26ca42c1b03d7a8c20b55ed87d2f0ce31e47" exitCode=60 Oct 30 01:15:19.851735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.848770 4517 scope.go:122] "RemoveContainer" containerID="36c75a27c02292d15c2c0b26149eac211403e90179cf67ec84c014b4a51e1db2" Oct 30 01:15:19.873078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.873046 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw"] Oct 30 01:15:19.950294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.945652 4517 generic.go:356] "Generic (PLEG): container finished" podID="a09d6932-71e7-4876-81a8-8964658c2b98" containerID="99fafa040cb40f4ca45c9e933b9c42ca6104882ca8c57d993c37424f4210806a" exitCode=60 Oct 30 01:15:19.950294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.945745 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" event={"ID":"a09d6932-71e7-4876-81a8-8964658c2b98","Type":"ContainerDied","Data":"99fafa040cb40f4ca45c9e933b9c42ca6104882ca8c57d993c37424f4210806a"} Oct 30 01:15:19.965376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:19.964103 4517 scope.go:122] "RemoveContainer" containerID="99fafa040cb40f4ca45c9e933b9c42ca6104882ca8c57d993c37424f4210806a" Oct 30 01:15:19.965376 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:19.964612 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm_odh-ai-gateway-infra(a09d6932-71e7-4876-81a8-8964658c2b98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" Oct 30 01:15:20.157018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:20.151000 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm"] Oct 30 01:15:20.318321 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:20.318284 4517 scope.go:122] "RemoveContainer" containerID="c2540c3f2bb38126c503ff235c0b4a9a93ecde372429f013f13227f473f56bf6" Oct 30 01:15:20.379738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:20.373653 4517 scope.go:122] "RemoveContainer" containerID="b792999e89f7c0e9b372faa952beaa6aca1d01c2218ec86bbd5bdf0ef0bc39c3" Oct 30 01:15:20.399550 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:20.396746 4517 scope.go:122] "RemoveContainer" containerID="df3a0af876ca00afa07b439131e6bcab8952632d613c9140845fa6ed3f7d61e3" Oct 30 01:15:20.543069 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:20.491279 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:20.825134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:20.813822 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:15:20.852451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:20.852391 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:15:20.971604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:20.955956 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:15:21.091929 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.091880 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:15:21.313797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.306570 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:21.408771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.394396 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:15:21.473928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.473874 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:15:21.523129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.523009 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 01:15:21.776379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.530934 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" podUID="078c1897-20b9-49d7-8096-d60f593e419e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.159:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:15:21.776633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.597671 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:15:21.777167 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:21.632521 4517 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.344s" Oct 30 01:15:21.777418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.755325 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:15:21.844915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.840813 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:15:21.844915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.841689 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 01:15:22.032999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:21.999244 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 01:15:22.043216 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.038673 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:22.064633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.064583 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh"] Oct 30 01:15:22.075972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.075934 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" event={"ID":"953fecbb-4d1a-4626-ac60-dd0825d89a5e","Type":"ContainerStarted","Data":"f4229e64b818a27e2abd95b9aeb03c5a1146cf291d065580df3f8114cac78339"} Oct 30 01:15:22.077352 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.077322 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:15:22.100605 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.098071 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w"] Oct 30 01:15:22.126721 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.123079 4517 generic.go:356] "Generic (PLEG): container finished" podID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerID="351aca377c894874e967f5a4f559e00dea17e9bf800093f998911f76a69619b1" exitCode=60 Oct 30 01:15:22.126721 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.123298 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" event={"ID":"98219cb4-5af9-4242-8524-12cc774bd3fc","Type":"ContainerDied","Data":"351aca377c894874e967f5a4f559e00dea17e9bf800093f998911f76a69619b1"} Oct 30 01:15:22.126721 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.123370 4517 scope.go:122] "RemoveContainer" containerID="c7131ce93112f7460913c53ba353e44005bb94ae3594d5ce09170a81009d3a22" Oct 30 01:15:22.136658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.136580 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34effdd-fdf2-49f3-b971-b12b0315257b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34effdd-fdf2-49f3-b971-b12b0315257b-maas-api-service-ca\") pod \"f34effdd-fdf2-49f3-b971-b12b0315257b\" (UID: \"f34effdd-fdf2-49f3-b971-b12b0315257b\") " Oct 30 01:15:22.137567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.137393 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34effdd-fdf2-49f3-b971-b12b0315257b-maas-api-service-ca" pod "f34effdd-fdf2-49f3-b971-b12b0315257b" (UID: "f34effdd-fdf2-49f3-b971-b12b0315257b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.138821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.138257 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34effdd-fdf2-49f3-b971-b12b0315257b-kube-api-access-bmb6x\" (UniqueName: \"kubernetes.io/projected/f34effdd-fdf2-49f3-b971-b12b0315257b-kube-api-access-bmb6x\") pod \"f34effdd-fdf2-49f3-b971-b12b0315257b\" (UID: \"f34effdd-fdf2-49f3-b971-b12b0315257b\") " Oct 30 01:15:22.139730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.139350 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34effdd-fdf2-49f3-b971-b12b0315257b-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.142385 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.141955 4517 generic.go:356] "Generic (PLEG): container finished" podID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerID="68ede6897bdac3d3c28f466cefc2bda57961db6711a785be23b5ce249e5bbd28" exitCode=60 Oct 30 01:15:22.142385 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.142098 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" event={"ID":"d051c589-6cf9-4544-896f-5cc2d09feee3","Type":"ContainerDied","Data":"68ede6897bdac3d3c28f466cefc2bda57961db6711a785be23b5ce249e5bbd28"} Oct 30 01:15:22.144841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.144744 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34effdd-fdf2-49f3-b971-b12b0315257b-kube-api-access-bmb6x" pod "f34effdd-fdf2-49f3-b971-b12b0315257b" (UID: "f34effdd-fdf2-49f3-b971-b12b0315257b"). InnerVolumeSpecName "kube-api-access-bmb6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.148782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.147536 4517 generic.go:356] "Generic (PLEG): container finished" podID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerID="1ccc1046bcebdc597cda7824f9c154691558ab822dd60340546c228b941a0156" exitCode=60 Oct 30 01:15:22.148782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.147673 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" event={"ID":"1d7826a2-7045-4db6-9952-fbc16ddc5e22","Type":"ContainerDied","Data":"1ccc1046bcebdc597cda7824f9c154691558ab822dd60340546c228b941a0156"} Oct 30 01:15:22.148782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.148245 4517 scope.go:122] "RemoveContainer" containerID="1ccc1046bcebdc597cda7824f9c154691558ab822dd60340546c228b941a0156" Oct 30 01:15:22.149209 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:22.149131 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf_odh-ai-gateway-infra(1d7826a2-7045-4db6-9952-fbc16ddc5e22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" Oct 30 01:15:22.152053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.152022 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" event={"ID":"7bdc7cb4-c3d9-4d7f-917f-e427552083f1","Type":"ContainerStarted","Data":"c7a2b757683a244f2e4ad381a57b78ce5b8b24b1d956e7480e3d899b5f35db4e"} Oct 30 01:15:22.152819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.152798 4517 scope.go:122] "RemoveContainer" containerID="c7a2b757683a244f2e4ad381a57b78ce5b8b24b1d956e7480e3d899b5f35db4e" Oct 30 01:15:22.153908 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:22.153882 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw_odh-ai-gateway-infra(7bdc7cb4-c3d9-4d7f-917f-e427552083f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" Oct 30 01:15:22.154056 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:22.152847 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdc7cb4_c3d9_4d7f_917f_e427552083f1.slice/crio-conmon-c7a2b757683a244f2e4ad381a57b78ce5b8b24b1d956e7480e3d899b5f35db4e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:22.183035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.183012 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf"] Oct 30 01:15:22.193685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.193590 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:22.259350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.251598 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f77bcc-c251-49e5-a309-9dede3b87810-kube-api-access-95n7l\" (UniqueName: \"kubernetes.io/projected/10f77bcc-c251-49e5-a309-9dede3b87810-kube-api-access-95n7l\") pod \"10f77bcc-c251-49e5-a309-9dede3b87810\" (UID: \"10f77bcc-c251-49e5-a309-9dede3b87810\") " Oct 30 01:15:22.259350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.251920 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f77bcc-c251-49e5-a309-9dede3b87810-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f77bcc-c251-49e5-a309-9dede3b87810-maas-api-service-ca\") pod \"10f77bcc-c251-49e5-a309-9dede3b87810\" (UID: \"10f77bcc-c251-49e5-a309-9dede3b87810\") " Oct 30 01:15:22.259350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.256372 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f77bcc-c251-49e5-a309-9dede3b87810-maas-api-service-ca" pod "10f77bcc-c251-49e5-a309-9dede3b87810" (UID: "10f77bcc-c251-49e5-a309-9dede3b87810"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.274322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.271543 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmb6x\" (UniqueName: \"kubernetes.io/projected/f34effdd-fdf2-49f3-b971-b12b0315257b-kube-api-access-bmb6x\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.274322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.271875 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f77bcc-c251-49e5-a309-9dede3b87810-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.306241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.287180 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f77bcc-c251-49e5-a309-9dede3b87810-kube-api-access-95n7l" pod "10f77bcc-c251-49e5-a309-9dede3b87810" (UID: "10f77bcc-c251-49e5-a309-9dede3b87810"). InnerVolumeSpecName "kube-api-access-95n7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.320947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.320912 4517 scope.go:122] "RemoveContainer" containerID="12f58e56ca711819f368e1d54a9c212ca7a4fd141f71e550cea20f48b83857f2" Oct 30 01:15:22.423499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.423474 4517 scope.go:122] "RemoveContainer" containerID="5d81bb8c2352479594d39197f843f786bcaaf5c81e6c10a93d8624542db9ed1e" Oct 30 01:15:22.427545 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.427512 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95n7l\" (UniqueName: \"kubernetes.io/projected/10f77bcc-c251-49e5-a309-9dede3b87810-kube-api-access-95n7l\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.432810 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.432784 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:22.544247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.544212 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:22.616178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.616151 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:22.621784 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.621756 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:22.646746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.645975 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09d6932-71e7-4876-81a8-8964658c2b98-kube-api-access-fvlvb\" (UniqueName: \"kubernetes.io/projected/a09d6932-71e7-4876-81a8-8964658c2b98-kube-api-access-fvlvb\") pod \"a09d6932-71e7-4876-81a8-8964658c2b98\" (UID: \"a09d6932-71e7-4876-81a8-8964658c2b98\") " Oct 30 01:15:22.646746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.646140 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09d6932-71e7-4876-81a8-8964658c2b98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09d6932-71e7-4876-81a8-8964658c2b98-maas-api-service-ca\") pod \"a09d6932-71e7-4876-81a8-8964658c2b98\" (UID: \"a09d6932-71e7-4876-81a8-8964658c2b98\") " Oct 30 01:15:22.646746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.646221 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d051c589-6cf9-4544-896f-5cc2d09feee3-kube-api-access-646cf\" (UniqueName: \"kubernetes.io/projected/d051c589-6cf9-4544-896f-5cc2d09feee3-kube-api-access-646cf\") pod \"d051c589-6cf9-4544-896f-5cc2d09feee3\" (UID: \"d051c589-6cf9-4544-896f-5cc2d09feee3\") " Oct 30 01:15:22.646746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.646304 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d051c589-6cf9-4544-896f-5cc2d09feee3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d051c589-6cf9-4544-896f-5cc2d09feee3-maas-api-service-ca\") pod \"d051c589-6cf9-4544-896f-5cc2d09feee3\" (UID: \"d051c589-6cf9-4544-896f-5cc2d09feee3\") " Oct 30 01:15:22.646746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.646346 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98219cb4-5af9-4242-8524-12cc774bd3fc-kube-api-access-j6rq9\" (UniqueName: \"kubernetes.io/projected/98219cb4-5af9-4242-8524-12cc774bd3fc-kube-api-access-j6rq9\") pod \"98219cb4-5af9-4242-8524-12cc774bd3fc\" (UID: \"98219cb4-5af9-4242-8524-12cc774bd3fc\") " Oct 30 01:15:22.646746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.646396 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f855240f-d1f1-47c1-b8e4-d155a4ee590f-kube-api-access-l5sgc\" (UniqueName: \"kubernetes.io/projected/f855240f-d1f1-47c1-b8e4-d155a4ee590f-kube-api-access-l5sgc\") pod \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\" (UID: \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\") " Oct 30 01:15:22.646746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.646499 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98219cb4-5af9-4242-8524-12cc774bd3fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98219cb4-5af9-4242-8524-12cc774bd3fc-maas-api-service-ca\") pod \"98219cb4-5af9-4242-8524-12cc774bd3fc\" (UID: \"98219cb4-5af9-4242-8524-12cc774bd3fc\") " Oct 30 01:15:22.646746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.646525 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f855240f-d1f1-47c1-b8e4-d155a4ee590f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855240f-d1f1-47c1-b8e4-d155a4ee590f-maas-api-service-ca\") pod \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\" (UID: \"f855240f-d1f1-47c1-b8e4-d155a4ee590f\") " Oct 30 01:15:22.659849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.647407 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d051c589-6cf9-4544-896f-5cc2d09feee3-maas-api-service-ca" pod "d051c589-6cf9-4544-896f-5cc2d09feee3" (UID: "d051c589-6cf9-4544-896f-5cc2d09feee3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.659849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.647857 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f855240f-d1f1-47c1-b8e4-d155a4ee590f-maas-api-service-ca" pod "f855240f-d1f1-47c1-b8e4-d155a4ee590f" (UID: "f855240f-d1f1-47c1-b8e4-d155a4ee590f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.659849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.652093 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98219cb4-5af9-4242-8524-12cc774bd3fc-maas-api-service-ca" pod "98219cb4-5af9-4242-8524-12cc774bd3fc" (UID: "98219cb4-5af9-4242-8524-12cc774bd3fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.659849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.652403 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09d6932-71e7-4876-81a8-8964658c2b98-maas-api-service-ca" pod "a09d6932-71e7-4876-81a8-8964658c2b98" (UID: "a09d6932-71e7-4876-81a8-8964658c2b98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.664214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.664170 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f855240f-d1f1-47c1-b8e4-d155a4ee590f-kube-api-access-l5sgc" pod "f855240f-d1f1-47c1-b8e4-d155a4ee590f" (UID: "f855240f-d1f1-47c1-b8e4-d155a4ee590f"). InnerVolumeSpecName "kube-api-access-l5sgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.667223 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.667179 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d051c589-6cf9-4544-896f-5cc2d09feee3-kube-api-access-646cf" pod "d051c589-6cf9-4544-896f-5cc2d09feee3" (UID: "d051c589-6cf9-4544-896f-5cc2d09feee3"). InnerVolumeSpecName "kube-api-access-646cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.668215 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.668175 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98219cb4-5af9-4242-8524-12cc774bd3fc-kube-api-access-j6rq9" pod "98219cb4-5af9-4242-8524-12cc774bd3fc" (UID: "98219cb4-5af9-4242-8524-12cc774bd3fc"). InnerVolumeSpecName "kube-api-access-j6rq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.668814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.668783 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09d6932-71e7-4876-81a8-8964658c2b98-kube-api-access-fvlvb" pod "a09d6932-71e7-4876-81a8-8964658c2b98" (UID: "a09d6932-71e7-4876-81a8-8964658c2b98"). InnerVolumeSpecName "kube-api-access-fvlvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.749097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.748935 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvlvb\" (UniqueName: \"kubernetes.io/projected/a09d6932-71e7-4876-81a8-8964658c2b98-kube-api-access-fvlvb\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.749097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.748977 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09d6932-71e7-4876-81a8-8964658c2b98-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.749097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.748993 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646cf\" (UniqueName: \"kubernetes.io/projected/d051c589-6cf9-4544-896f-5cc2d09feee3-kube-api-access-646cf\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.749097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.749011 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d051c589-6cf9-4544-896f-5cc2d09feee3-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.749097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.749026 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6rq9\" (UniqueName: \"kubernetes.io/projected/98219cb4-5af9-4242-8524-12cc774bd3fc-kube-api-access-j6rq9\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.749097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.749041 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5sgc\" (UniqueName: \"kubernetes.io/projected/f855240f-d1f1-47c1-b8e4-d155a4ee590f-kube-api-access-l5sgc\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.749097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.749055 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98219cb4-5af9-4242-8524-12cc774bd3fc-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.749097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.749069 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855240f-d1f1-47c1-b8e4-d155a4ee590f-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.937435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:22.937319 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:23.191555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.191436 4517 generic.go:356] "Generic (PLEG): container finished" podID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerID="c7a2b757683a244f2e4ad381a57b78ce5b8b24b1d956e7480e3d899b5f35db4e" exitCode=60 Oct 30 01:15:23.191555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.191509 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" event={"ID":"7bdc7cb4-c3d9-4d7f-917f-e427552083f1","Type":"ContainerDied","Data":"c7a2b757683a244f2e4ad381a57b78ce5b8b24b1d956e7480e3d899b5f35db4e"} Oct 30 01:15:23.191826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.191628 4517 scope.go:122] "RemoveContainer" containerID="58ba8fb4044296f84e4c163313d79974d885cee9dd5d6b766407f5f2c665faa0" Oct 30 01:15:23.192222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.192203 4517 scope.go:122] "RemoveContainer" containerID="c7a2b757683a244f2e4ad381a57b78ce5b8b24b1d956e7480e3d899b5f35db4e" Oct 30 01:15:23.192685 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:23.192659 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw_odh-ai-gateway-infra(7bdc7cb4-c3d9-4d7f-917f-e427552083f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" Oct 30 01:15:23.195948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.195823 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" event={"ID":"10f77bcc-c251-49e5-a309-9dede3b87810","Type":"ContainerDied","Data":"47421adbea8dac55ba6f2c7dbccc435f7846c359872106f4b59605b5f6890d4e"} Oct 30 01:15:23.196033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.195957 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs" Oct 30 01:15:23.199168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.198892 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" event={"ID":"a09d6932-71e7-4876-81a8-8964658c2b98","Type":"ContainerDied","Data":"987acf0120012cb8aceac3b14020cab3dcef38c85021ae4fd21e323d829ed2c2"} Oct 30 01:15:23.199168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.199005 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm" Oct 30 01:15:23.208772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.208465 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" event={"ID":"f34effdd-fdf2-49f3-b971-b12b0315257b","Type":"ContainerDied","Data":"0559cecf0d122f0fb6335a848ce06d13f848e35752fd4911e5e7b465f9ae4081"} Oct 30 01:15:23.209681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.209060 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4" Oct 30 01:15:23.213522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.213479 4517 generic.go:356] "Generic (PLEG): container finished" podID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerID="f4229e64b818a27e2abd95b9aeb03c5a1146cf291d065580df3f8114cac78339" exitCode=60 Oct 30 01:15:23.213721 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.213545 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" event={"ID":"953fecbb-4d1a-4626-ac60-dd0825d89a5e","Type":"ContainerDied","Data":"f4229e64b818a27e2abd95b9aeb03c5a1146cf291d065580df3f8114cac78339"} Oct 30 01:15:23.214155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.214133 4517 scope.go:122] "RemoveContainer" containerID="f4229e64b818a27e2abd95b9aeb03c5a1146cf291d065580df3f8114cac78339" Oct 30 01:15:23.214662 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:23.214625 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6_odh-ai-gateway-infra(953fecbb-4d1a-4626-ac60-dd0825d89a5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" Oct 30 01:15:23.215569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.215443 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" Oct 30 01:15:23.215569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.215470 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw" event={"ID":"f855240f-d1f1-47c1-b8e4-d155a4ee590f","Type":"ContainerDied","Data":"c741d57fa0b4773221c703e78055d1b7bca0b1a606ca3919918f64cb5419f536"} Oct 30 01:15:23.217035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.216881 4517 scope.go:122] "RemoveContainer" containerID="e96d8e1daf9cbe88e342fd81009f26ca42c1b03d7a8c20b55ed87d2f0ce31e47" Oct 30 01:15:23.218461 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.218371 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" Oct 30 01:15:23.218461 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.218425 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh" event={"ID":"98219cb4-5af9-4242-8524-12cc774bd3fc","Type":"ContainerDied","Data":"c16a7a9d0947b91a6cffe062713e0b1ca7fa0550efba38ffa0ffd6920fc0dc68"} Oct 30 01:15:23.221412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.221378 4517 generic.go:356] "Generic (PLEG): container finished" podID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerID="321a94e1486dac3522c395c91e0b190cefc078a2d0cf15b1808efa3e0e469217" exitCode=60 Oct 30 01:15:23.221583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.221447 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" event={"ID":"e7837d7d-e6f2-45cc-9258-d29fdfe6764f","Type":"ContainerDied","Data":"321a94e1486dac3522c395c91e0b190cefc078a2d0cf15b1808efa3e0e469217"} Oct 30 01:15:23.222020 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.221996 4517 scope.go:122] "RemoveContainer" containerID="321a94e1486dac3522c395c91e0b190cefc078a2d0cf15b1808efa3e0e469217" Oct 30 01:15:23.222450 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:23.222421 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd_odh-ai-gateway-infra(e7837d7d-e6f2-45cc-9258-d29fdfe6764f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" Oct 30 01:15:23.232516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.231490 4517 generic.go:356] "Generic (PLEG): container finished" podID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerID="1768fe6ba101eb4b510d4688c415003a3ccf72ed638965220a10ddfcce2dbb8d" exitCode=60 Oct 30 01:15:23.232516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.231550 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" event={"ID":"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8","Type":"ContainerDied","Data":"1768fe6ba101eb4b510d4688c415003a3ccf72ed638965220a10ddfcce2dbb8d"} Oct 30 01:15:23.232516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.232105 4517 scope.go:122] "RemoveContainer" containerID="1768fe6ba101eb4b510d4688c415003a3ccf72ed638965220a10ddfcce2dbb8d" Oct 30 01:15:23.232516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.232134 4517 scope.go:122] "RemoveContainer" containerID="99fafa040cb40f4ca45c9e933b9c42ca6104882ca8c57d993c37424f4210806a" Oct 30 01:15:23.232516 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:23.232470 4517 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686_odh-ai-gateway-infra(fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" Oct 30 01:15:23.234188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.234153 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" event={"ID":"d051c589-6cf9-4544-896f-5cc2d09feee3","Type":"ContainerDied","Data":"16b73b68b0918acd519f5c39165ca175087585cf230024e6594c5435411d1c91"} Oct 30 01:15:23.234274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.234251 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w" Oct 30 01:15:23.248865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.248782 4517 scope.go:122] "RemoveContainer" containerID="cb453543115e4fc8cc193e83557154c8ac33861d049eed07636b0089c7900257" Oct 30 01:15:23.259790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.258411 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6"] Oct 30 01:15:23.292773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.292687 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4"] Oct 30 01:15:23.313435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.310724 4517 scope.go:122] "RemoveContainer" containerID="48eddee4e6d086b6f483e9cf12166c5e85afeda3b321948c0f7ebde990b8f362" Oct 30 01:15:23.319560 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.319491 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d01638-29888715-dwlj4"] Oct 30 01:15:23.348436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.348412 4517 scope.go:122] "RemoveContainer" containerID="267778e97920af6a5b0dfed142c5fecd62d1d62f026f10cfd913ca523e29ab1f" Oct 30 01:15:23.356036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.355993 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs"] Oct 30 01:15:23.374537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.374221 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-320045-29888715-282rs"] Oct 30 01:15:23.389973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.389941 4517 scope.go:122] "RemoveContainer" containerID="351aca377c894874e967f5a4f559e00dea17e9bf800093f998911f76a69619b1" Oct 30 01:15:23.405400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.405277 4517 scope.go:122] "RemoveContainer" containerID="c2540c3f2bb38126c503ff235c0b4a9a93ecde372429f013f13227f473f56bf6" Oct 30 01:15:23.432555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.432398 4517 scope.go:122] "RemoveContainer" containerID="b792999e89f7c0e9b372faa952beaa6aca1d01c2218ec86bbd5bdf0ef0bc39c3" Oct 30 01:15:23.435180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.435151 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw"] Oct 30 01:15:23.435382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.435369 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f69104-29888715-228qw"] Oct 30 01:15:23.451792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.451769 4517 scope.go:122] "RemoveContainer" containerID="68ede6897bdac3d3c28f466cefc2bda57961db6711a785be23b5ce249e5bbd28" Oct 30 01:15:23.496333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.496293 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw"] Oct 30 01:15:23.507639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.507605 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm"] Oct 30 01:15:23.512475 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.512432 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:23.512651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.512603 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c4f6f-29888715-vhncm"] Oct 30 01:15:23.597630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.597588 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w"] Oct 30 01:15:23.603261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.603217 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8e24a5-29888715-gbd2w"] Oct 30 01:15:23.614935 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.614732 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:23.627965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.626854 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7826a2-7045-4db6-9952-fbc16ddc5e22-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7826a2-7045-4db6-9952-fbc16ddc5e22-maas-api-service-ca\") pod \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\" (UID: \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\") " Oct 30 01:15:23.627965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.627129 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7826a2-7045-4db6-9952-fbc16ddc5e22-kube-api-access-7tt2b\" (UniqueName: \"kubernetes.io/projected/1d7826a2-7045-4db6-9952-fbc16ddc5e22-kube-api-access-7tt2b\") pod \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\" (UID: \"1d7826a2-7045-4db6-9952-fbc16ddc5e22\") " Oct 30 01:15:23.632119 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.631929 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7826a2-7045-4db6-9952-fbc16ddc5e22-kube-api-access-7tt2b" pod "1d7826a2-7045-4db6-9952-fbc16ddc5e22" (UID: "1d7826a2-7045-4db6-9952-fbc16ddc5e22"). InnerVolumeSpecName "kube-api-access-7tt2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:23.632955 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.632126 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7826a2-7045-4db6-9952-fbc16ddc5e22-maas-api-service-ca" pod "1d7826a2-7045-4db6-9952-fbc16ddc5e22" (UID: "1d7826a2-7045-4db6-9952-fbc16ddc5e22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:23.723042 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.722933 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh"] Oct 30 01:15:23.729947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.729904 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-89a676-29888715-m24qh"] Oct 30 01:15:23.730163 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.730082 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tt2b\" (UniqueName: \"kubernetes.io/projected/1d7826a2-7045-4db6-9952-fbc16ddc5e22-kube-api-access-7tt2b\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:23.730163 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:23.730111 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7826a2-7045-4db6-9952-fbc16ddc5e22-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:24.240430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.240397 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" path="/var/lib/kubelet/pods/10f77bcc-c251-49e5-a309-9dede3b87810/volumes" Oct 30 01:15:24.240948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.240933 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" path="/var/lib/kubelet/pods/98219cb4-5af9-4242-8524-12cc774bd3fc/volumes" Oct 30 01:15:24.241392 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.241377 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" path="/var/lib/kubelet/pods/a09d6932-71e7-4876-81a8-8964658c2b98/volumes" Oct 30 01:15:24.242211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.242193 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" path="/var/lib/kubelet/pods/d051c589-6cf9-4544-896f-5cc2d09feee3/volumes" Oct 30 01:15:24.242671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.242649 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" path="/var/lib/kubelet/pods/f34effdd-fdf2-49f3-b971-b12b0315257b/volumes" Oct 30 01:15:24.243094 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.243080 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" path="/var/lib/kubelet/pods/f855240f-d1f1-47c1-b8e4-d155a4ee590f/volumes" Oct 30 01:15:24.274659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.272995 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" event={"ID":"1d7826a2-7045-4db6-9952-fbc16ddc5e22","Type":"ContainerDied","Data":"61777ce3b497776d94ea1b155aa31768ebd71645cc353ac0ae2e45b2f13787f7"} Oct 30 01:15:24.274659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.273074 4517 scope.go:122] "RemoveContainer" containerID="1ccc1046bcebdc597cda7824f9c154691558ab822dd60340546c228b941a0156" Oct 30 01:15:24.274659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.273253 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf" Oct 30 01:15:24.332971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.332922 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf"] Oct 30 01:15:24.340483 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.340443 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-beb8d5-29888715-rdjzf"] Oct 30 01:15:24.504497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.504398 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:24.570892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.570832 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd"] Oct 30 01:15:24.619755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.619285 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:24.673245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.672552 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-maas-api-service-ca\") pod \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\" (UID: \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\") " Oct 30 01:15:24.673245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.672675 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-kube-api-access-kqt8t\" (UniqueName: \"kubernetes.io/projected/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-kube-api-access-kqt8t\") pod \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\" (UID: \"7bdc7cb4-c3d9-4d7f-917f-e427552083f1\") " Oct 30 01:15:24.673245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.673069 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-maas-api-service-ca" pod "7bdc7cb4-c3d9-4d7f-917f-e427552083f1" (UID: "7bdc7cb4-c3d9-4d7f-917f-e427552083f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:24.673606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.673578 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:24.676221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.676157 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-kube-api-access-kqt8t" pod "7bdc7cb4-c3d9-4d7f-917f-e427552083f1" (UID: "7bdc7cb4-c3d9-4d7f-917f-e427552083f1"). InnerVolumeSpecName "kube-api-access-kqt8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:24.701382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.701344 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686"] Oct 30 01:15:24.775665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.775235 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953fecbb-4d1a-4626-ac60-dd0825d89a5e-kube-api-access-2dswp\" (UniqueName: \"kubernetes.io/projected/953fecbb-4d1a-4626-ac60-dd0825d89a5e-kube-api-access-2dswp\") pod \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\" (UID: \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\") " Oct 30 01:15:24.776784 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.776448 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953fecbb-4d1a-4626-ac60-dd0825d89a5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953fecbb-4d1a-4626-ac60-dd0825d89a5e-maas-api-service-ca\") pod \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\" (UID: \"953fecbb-4d1a-4626-ac60-dd0825d89a5e\") " Oct 30 01:15:24.777733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.777393 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqt8t\" (UniqueName: \"kubernetes.io/projected/7bdc7cb4-c3d9-4d7f-917f-e427552083f1-kube-api-access-kqt8t\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:24.777867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.777803 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953fecbb-4d1a-4626-ac60-dd0825d89a5e-maas-api-service-ca" pod "953fecbb-4d1a-4626-ac60-dd0825d89a5e" (UID: "953fecbb-4d1a-4626-ac60-dd0825d89a5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:24.781969 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.781915 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953fecbb-4d1a-4626-ac60-dd0825d89a5e-kube-api-access-2dswp" pod "953fecbb-4d1a-4626-ac60-dd0825d89a5e" (UID: "953fecbb-4d1a-4626-ac60-dd0825d89a5e"). InnerVolumeSpecName "kube-api-access-2dswp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:24.792218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.792188 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:24.891733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.884334 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-maas-api-service-ca\") pod \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\" (UID: \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\") " Oct 30 01:15:24.907690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.907649 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-maas-api-service-ca" pod "e7837d7d-e6f2-45cc-9258-d29fdfe6764f" (UID: "e7837d7d-e6f2-45cc-9258-d29fdfe6764f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:24.907962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.907932 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-kube-api-access-5sn9c\" (UniqueName: \"kubernetes.io/projected/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-kube-api-access-5sn9c\") pod \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\" (UID: \"e7837d7d-e6f2-45cc-9258-d29fdfe6764f\") " Oct 30 01:15:24.908771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.908687 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dswp\" (UniqueName: \"kubernetes.io/projected/953fecbb-4d1a-4626-ac60-dd0825d89a5e-kube-api-access-2dswp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:24.908771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.908731 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953fecbb-4d1a-4626-ac60-dd0825d89a5e-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:24.908771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.908745 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:24.911159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.911133 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-kube-api-access-5sn9c" pod "e7837d7d-e6f2-45cc-9258-d29fdfe6764f" (UID: "e7837d7d-e6f2-45cc-9258-d29fdfe6764f"). InnerVolumeSpecName "kube-api-access-5sn9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:24.929508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:24.929483 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:25.009868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.009823 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sn9c\" (UniqueName: \"kubernetes.io/projected/e7837d7d-e6f2-45cc-9258-d29fdfe6764f-kube-api-access-5sn9c\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:25.094322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.094276 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:25.111150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.111113 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-kube-api-access-47rzj\" (UniqueName: \"kubernetes.io/projected/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-kube-api-access-47rzj\") pod \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\" (UID: \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\") " Oct 30 01:15:25.111405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.111389 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-maas-api-service-ca\") pod \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\" (UID: \"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8\") " Oct 30 01:15:25.111944 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.111912 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-maas-api-service-ca" pod "fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" (UID: "fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:25.112084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.112068 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-maas-api-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:25.113962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.113931 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-kube-api-access-47rzj" pod "fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" (UID: "fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8"). InnerVolumeSpecName "kube-api-access-47rzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:25.214029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.213979 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47rzj\" (UniqueName: \"kubernetes.io/projected/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8-kube-api-access-47rzj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:25.282617 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.282578 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" event={"ID":"953fecbb-4d1a-4626-ac60-dd0825d89a5e","Type":"ContainerDied","Data":"ed0f3dca4fe5e2c7a392fe2767f9f8c739ea7de3a6bcc0a6d159a9fdfe3e5b4a"} Oct 30 01:15:25.282858 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.282589 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6" Oct 30 01:15:25.282858 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.282640 4517 scope.go:122] "RemoveContainer" containerID="f4229e64b818a27e2abd95b9aeb03c5a1146cf291d065580df3f8114cac78339" Oct 30 01:15:25.284829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.284781 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" Oct 30 01:15:25.285289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.284793 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd" event={"ID":"e7837d7d-e6f2-45cc-9258-d29fdfe6764f","Type":"ContainerDied","Data":"f8e0f0454e0cd2d6e561012f14847b694c64ff771043916301b3927eb1de0a99"} Oct 30 01:15:25.337007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.334200 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" event={"ID":"fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8","Type":"ContainerDied","Data":"507170f7c27f8b7f3271f38c8337cac95fe2233bee60fdc02e5088b25fdbc891"} Oct 30 01:15:25.337007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.334342 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686" Oct 30 01:15:25.351427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.350978 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" event={"ID":"7bdc7cb4-c3d9-4d7f-917f-e427552083f1","Type":"ContainerDied","Data":"fdd19c6dfefe3cbd78b6dde7550b8be0eabbaf2d50b1a8768a2444c306b18e3e"} Oct 30 01:15:25.351427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.351125 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw" Oct 30 01:15:25.365769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.362354 4517 scope.go:122] "RemoveContainer" containerID="321a94e1486dac3522c395c91e0b190cefc078a2d0cf15b1808efa3e0e469217" Oct 30 01:15:25.369882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.369838 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd"] Oct 30 01:15:25.379084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.377875 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f91e87-29888715-n2vtd"] Oct 30 01:15:25.388657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.388628 4517 scope.go:122] "RemoveContainer" containerID="1768fe6ba101eb4b510d4688c415003a3ccf72ed638965220a10ddfcce2dbb8d" Oct 30 01:15:25.401450 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.401430 4517 scope.go:122] "RemoveContainer" containerID="c7a2b757683a244f2e4ad381a57b78ce5b8b24b1d956e7480e3d899b5f35db4e" Oct 30 01:15:25.418084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.418049 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6"] Oct 30 01:15:25.423854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.423817 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-60e8f4-29888715-ktrf6"] Oct 30 01:15:25.467557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.467511 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw"] Oct 30 01:15:25.473338 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.473308 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888715-ntfzw"] Oct 30 01:15:25.503404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.503372 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686"] Oct 30 01:15:25.509137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:25.509109 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888715-ks686"] Oct 30 01:15:26.240313 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:26.240275 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" path="/var/lib/kubelet/pods/1d7826a2-7045-4db6-9952-fbc16ddc5e22/volumes" Oct 30 01:15:26.240810 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:26.240794 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" path="/var/lib/kubelet/pods/7bdc7cb4-c3d9-4d7f-917f-e427552083f1/volumes" Oct 30 01:15:26.241222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:26.241208 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" path="/var/lib/kubelet/pods/953fecbb-4d1a-4626-ac60-dd0825d89a5e/volumes" Oct 30 01:15:26.242016 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:26.241998 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" path="/var/lib/kubelet/pods/e7837d7d-e6f2-45cc-9258-d29fdfe6764f/volumes" Oct 30 01:15:26.242408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:26.242395 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" path="/var/lib/kubelet/pods/fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8/volumes" Oct 30 01:15:26.823373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:26.823339 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:27.113030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:27.112916 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs"] Oct 30 01:15:27.127812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:27.127698 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:28.629607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:28.629527 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:30.556870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:30.556839 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:30.763319 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:30.763279 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:15:32.213426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:32.213375 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:32.331664 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:32.331576 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:32.561420 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:32.561244 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:32.562072 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:32.561862 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:33.773912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:33.773874 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:35.455244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:35.455203 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:37.139399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:37.139358 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:37.310624 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:37.310589 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:38.707726 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:38.707671 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:40.630408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:40.630350 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:40.764747 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:40.756130 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:40.879124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:40.879077 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:42.146902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.146858 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" podUID="078c1897-20b9-49d7-8096-d60f593e419e" containerName="main" containerID="cri-o://59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b" gracePeriod=45 Oct 30 01:15:42.455836 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:42.455793 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:42.482993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.482945 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:15:42.547641 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.547588 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:42.551764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.551671 4517 generic.go:356] "Generic (PLEG): container finished" podID="078c1897-20b9-49d7-8096-d60f593e419e" containerID="59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b" exitCode=0 Oct 30 01:15:42.551764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.551754 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" Oct 30 01:15:42.552044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.551934 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" event={"ID":"078c1897-20b9-49d7-8096-d60f593e419e","Type":"ContainerDied","Data":"59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b"} Oct 30 01:15:42.552044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.551976 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" event={"ID":"078c1897-20b9-49d7-8096-d60f593e419e","Type":"ContainerDied","Data":"1d5a0d2f041faceb59c81cca87820439ab8cd0d7ff9e1251c9651cda66524570"} Oct 30 01:15:42.552044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.552003 4517 scope.go:122] "RemoveContainer" containerID="59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b" Oct 30 01:15:42.563933 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.563899 4517 scope.go:122] "RemoveContainer" containerID="59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b" Oct 30 01:15:42.564394 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:42.564360 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b\": container with ID starting with 59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b not found: ID does not exist" containerID="59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b" Oct 30 01:15:42.564536 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.564403 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b"} err="failed to get container status \"59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b\": rpc error: code = NotFound desc = could not find container \"59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b\": container with ID starting with 59bf6d1b5f1d36736ce9c97d0de163173251ba916e5333ca43056ce5955bb12b not found: ID does not exist" Oct 30 01:15:42.578602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.578555 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078c1897-20b9-49d7-8096-d60f593e419e-kube-api-access-sgspj\" (UniqueName: \"kubernetes.io/projected/078c1897-20b9-49d7-8096-d60f593e419e-kube-api-access-sgspj\") pod \"078c1897-20b9-49d7-8096-d60f593e419e\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " Oct 30 01:15:42.578814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.578685 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-tmp-dir\") pod \"078c1897-20b9-49d7-8096-d60f593e419e\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " Oct 30 01:15:42.578814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.578789 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-dshm\") pod \"078c1897-20b9-49d7-8096-d60f593e419e\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " Oct 30 01:15:42.578963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.578915 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-model-cache\") pod \"078c1897-20b9-49d7-8096-d60f593e419e\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " Oct 30 01:15:42.579100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.579076 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-home\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-home\") pod \"078c1897-20b9-49d7-8096-d60f593e419e\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " Oct 30 01:15:42.579165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.579154 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078c1897-20b9-49d7-8096-d60f593e419e-tls-certs\" (UniqueName: \"kubernetes.io/secret/078c1897-20b9-49d7-8096-d60f593e419e-tls-certs\") pod \"078c1897-20b9-49d7-8096-d60f593e419e\" (UID: \"078c1897-20b9-49d7-8096-d60f593e419e\") " Oct 30 01:15:42.580672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.580251 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-model-cache" pod "078c1897-20b9-49d7-8096-d60f593e419e" (UID: "078c1897-20b9-49d7-8096-d60f593e419e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:42.580672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.580473 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-home" pod "078c1897-20b9-49d7-8096-d60f593e419e" (UID: "078c1897-20b9-49d7-8096-d60f593e419e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:42.580958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.580792 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-tmp-dir" pod "078c1897-20b9-49d7-8096-d60f593e419e" (UID: "078c1897-20b9-49d7-8096-d60f593e419e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:42.584962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.584807 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078c1897-20b9-49d7-8096-d60f593e419e-tls-certs" pod "078c1897-20b9-49d7-8096-d60f593e419e" (UID: "078c1897-20b9-49d7-8096-d60f593e419e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:15:42.584962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.585387 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-dshm" pod "078c1897-20b9-49d7-8096-d60f593e419e" (UID: "078c1897-20b9-49d7-8096-d60f593e419e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:42.645408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.645290 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078c1897-20b9-49d7-8096-d60f593e419e-kube-api-access-sgspj" pod "078c1897-20b9-49d7-8096-d60f593e419e" (UID: "078c1897-20b9-49d7-8096-d60f593e419e"). InnerVolumeSpecName "kube-api-access-sgspj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:42.681922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.681762 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgspj\" (UniqueName: \"kubernetes.io/projected/078c1897-20b9-49d7-8096-d60f593e419e-kube-api-access-sgspj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:42.681922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.681806 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:42.681922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.681819 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:42.681922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.681830 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:42.681922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.681843 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078c1897-20b9-49d7-8096-d60f593e419e-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:42.681922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.681854 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078c1897-20b9-49d7-8096-d60f593e419e-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:42.897613 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.897574 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs"] Oct 30 01:15:42.902227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:42.902188 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs"] Oct 30 01:15:43.455619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.455567 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-33e759/auth-test-model-33e759-kserve-64ff8f547c-nd2hs" podUID="078c1897-20b9-49d7-8096-d60f593e419e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.159:8000/ready\": context deadline exceeded" Oct 30 01:15:43.919942 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.919892 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7"] Oct 30 01:15:43.920780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920742 4517 cpu_manager.go:414] "containerMap: removing container" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerName="cleanup" Oct 30 01:15:43.920780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920769 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerName="cleanup" Oct 30 01:15:43.920780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920788 4517 cpu_manager.go:414] "containerMap: removing container" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920797 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920809 4517 cpu_manager.go:414] "containerMap: removing container" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920816 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920836 4517 cpu_manager.go:414] "containerMap: removing container" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920845 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920854 4517 cpu_manager.go:414] "containerMap: removing container" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920859 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920867 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920875 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920883 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920891 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920904 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920910 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920936 4517 cpu_manager.go:414] "containerMap: removing container" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920943 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920954 4517 cpu_manager.go:414] "containerMap: removing container" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920961 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920976 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920983 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.920992 4517 cpu_manager.go:414] "containerMap: removing container" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921000 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921010 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921016 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921027 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921034 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921043 4517 cpu_manager.go:414] "containerMap: removing container" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921051 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921063 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921070 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921080 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921086 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921099 4517 cpu_manager.go:414] "containerMap: removing container" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921106 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921115 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921122 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerName="cleanup" Oct 30 01:15:43.921110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921134 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921141 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921184 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921193 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921202 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921209 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921244 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921252 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921271 4517 cpu_manager.go:414] "containerMap: removing container" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921278 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921294 4517 cpu_manager.go:414] "containerMap: removing container" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921301 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921315 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921322 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921338 4517 cpu_manager.go:414] "containerMap: removing container" podUID="078c1897-20b9-49d7-8096-d60f593e419e" containerName="main" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921344 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078c1897-20b9-49d7-8096-d60f593e419e" containerName="main" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921352 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921360 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921369 4517 cpu_manager.go:414] "containerMap: removing container" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921374 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921405 4517 cpu_manager.go:414] "containerMap: removing container" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921412 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921427 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921432 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921661 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921676 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921688 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921700 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921731 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921739 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921748 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921755 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921762 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921770 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7826a2-7045-4db6-9952-fbc16ddc5e22" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921781 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921790 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921799 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921806 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d051c589-6cf9-4544-896f-5cc2d09feee3" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921812 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921823 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="078c1897-20b9-49d7-8096-d60f593e419e" containerName="main" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921830 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921837 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921849 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdc7cb4-c3d9-4d7f-917f-e427552083f1" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921858 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921867 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921875 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921884 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921892 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921904 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerName="cleanup" Oct 30 01:15:43.923027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.921910 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerName="cleanup" Oct 30 01:15:43.927484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.927454 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:43.936950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.936910 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb"] Oct 30 01:15:43.938020 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.937994 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerName="cleanup" Oct 30 01:15:43.938156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938022 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerName="cleanup" Oct 30 01:15:43.938156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938076 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerName="cleanup" Oct 30 01:15:43.938156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938084 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerName="cleanup" Oct 30 01:15:43.938156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938132 4517 cpu_manager.go:414] "containerMap: removing container" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerName="cleanup" Oct 30 01:15:43.938156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938140 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerName="cleanup" Oct 30 01:15:43.938400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938322 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0e9639-b2ab-450a-9cc0-0ce6b0894cc8" containerName="cleanup" Oct 30 01:15:43.938400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938341 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f77bcc-c251-49e5-a309-9dede3b87810" containerName="cleanup" Oct 30 01:15:43.938400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938351 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34effdd-fdf2-49f3-b971-b12b0315257b" containerName="cleanup" Oct 30 01:15:43.938400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938361 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7837d7d-e6f2-45cc-9258-d29fdfe6764f" containerName="cleanup" Oct 30 01:15:43.938400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938374 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f855240f-d1f1-47c1-b8e4-d155a4ee590f" containerName="cleanup" Oct 30 01:15:43.938400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938387 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09d6932-71e7-4876-81a8-8964658c2b98" containerName="cleanup" Oct 30 01:15:43.938400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938399 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="98219cb4-5af9-4242-8524-12cc774bd3fc" containerName="cleanup" Oct 30 01:15:43.938814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.938408 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="953fecbb-4d1a-4626-ac60-dd0825d89a5e" containerName="cleanup" Oct 30 01:15:43.942672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.942647 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:43.952753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.952555 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7"] Oct 30 01:15:43.994120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:43.994053 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb"] Oct 30 01:15:44.005540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.005492 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.005810 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.005554 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.005810 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.005680 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.005949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.005896 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.006008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.005962 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.006008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.005993 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.006121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006039 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmf8\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-kube-api-access-qqmf8\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.006175 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006135 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.006227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006201 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336252d7-b778-490b-93e8-7f2c4ca5db60-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.006293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006267 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-784sg\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-kube-api-access-784sg\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.006357 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006319 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.006416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006359 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.006540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006437 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.006540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006475 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.006540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006508 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.006731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006542 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.006731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006590 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.006731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.006664 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.108000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.107959 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.108222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108020 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336252d7-b778-490b-93e8-7f2c4ca5db60-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.108222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108053 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784sg\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-kube-api-access-784sg\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.108222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108084 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.108222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108119 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.108222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108177 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.108222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108199 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.108222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108218 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.108648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108552 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.108648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108607 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.108787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108639 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.108787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108670 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.109011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108786 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.109011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108817 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.109011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108873 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.109011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108898 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.109011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108925 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336252d7-b778-490b-93e8-7f2c4ca5db60-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.109011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108938 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.109011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.108947 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.109011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.109034 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.109689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.109131 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.109689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.109155 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.109689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.109164 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.109689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.109337 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.109689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.109375 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmf8\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-kube-api-access-qqmf8\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.109689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.109389 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.109689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.109436 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.109689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.109556 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.111152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.111069 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.111152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.111145 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.112009 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.111991 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.112268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.112239 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.133312 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.133276 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-784sg\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-kube-api-access-784sg\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.133674 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.133645 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.133792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.133680 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.135617 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.135591 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmf8\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-kube-api-access-qqmf8\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.237701 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.237614 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078c1897-20b9-49d7-8096-d60f593e419e" path="/var/lib/kubelet/pods/078c1897-20b9-49d7-8096-d60f593e419e/volumes" Oct 30 01:15:44.247029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.246996 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:44.291025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.290986 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:44.370830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.370669 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:44.609442 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.607839 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:15:44.609442 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.607933 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:15:44.609442 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.607981 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:15:44.615291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.615253 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7"] Oct 30 01:15:44.653961 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.653928 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb"] Oct 30 01:15:44.655651 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:15:44.655619 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336252d7_b778_490b_93e8_7f2c4ca5db60.slice/crio-0ac3fe19d3764f9b401847863f130c88f1cd34a4a288acc8beb8846ef8bf7689 WatchSource:0}: Error finding container 0ac3fe19d3764f9b401847863f130c88f1cd34a4a288acc8beb8846ef8bf7689: Status 404 returned error can't find the container with id 0ac3fe19d3764f9b401847863f130c88f1cd34a4a288acc8beb8846ef8bf7689 Oct 30 01:15:44.660064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.660010 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:15:44.660209 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.660111 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:15:44.660209 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:44.660162 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:15:45.624661 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:45.624612 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" event={"ID":"ee1086e2-a267-44c9-b5a8-f94c1aa59385","Type":"ContainerStarted","Data":"29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f"} Oct 30 01:15:45.625143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:45.624691 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" event={"ID":"ee1086e2-a267-44c9-b5a8-f94c1aa59385","Type":"ContainerStarted","Data":"e0daaa1146bfe360f555abbf91a5577d09528298cab51bb62c7c0a81611c9d61"} Oct 30 01:15:45.627369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:45.627336 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" event={"ID":"336252d7-b778-490b-93e8-7f2c4ca5db60","Type":"ContainerStarted","Data":"3a4f6c81a36a27695bf90cc2101a037c0109bba926a1bdddea78d49ccb98a143"} Oct 30 01:15:45.627555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:45.627389 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" event={"ID":"336252d7-b778-490b-93e8-7f2c4ca5db60","Type":"ContainerStarted","Data":"0ac3fe19d3764f9b401847863f130c88f1cd34a4a288acc8beb8846ef8bf7689"} Oct 30 01:15:45.721957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:45.721897 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" podStartSLOduration=2.721881617 podStartE2EDuration="2.721881617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:45.721752788 +0000 UTC m=+2483.633879928" watchObservedRunningTime="2026-10-30 01:15:45.721881617 +0000 UTC m=+2483.634008756" Oct 30 01:15:45.726399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:45.726342 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" podStartSLOduration=2.726324816 podStartE2EDuration="2.726324816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:45.679114998 +0000 UTC m=+2483.591242150" watchObservedRunningTime="2026-10-30 01:15:45.726324816 +0000 UTC m=+2483.638451955" Oct 30 01:15:46.174735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:46.158811 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:46.281751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:46.250796 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:46.281751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:46.268978 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 01:15:46.281751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:46.269043 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 01:15:46.296608 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:46.296571 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 01:15:46.296952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:46.296905 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 01:15:46.332297 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:46.332262 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:47.248277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:47.248223 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 01:15:47.248828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:47.248310 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 01:15:47.293736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:47.293626 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 01:15:47.293736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:47.293726 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 01:15:47.865738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:47.863047 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:48.247612 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:48.247504 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 01:15:48.247612 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:48.247581 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 01:15:48.293738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:48.293334 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 01:15:48.293738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:48.293400 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 01:15:49.247556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:49.247510 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 01:15:49.247899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:49.247871 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 01:15:49.293737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:49.293430 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 01:15:49.293737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:49.293499 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 01:15:49.753813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:49.752671 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:50.248046 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:50.247397 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 01:15:50.248046 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:50.247517 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 01:15:50.296759 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:50.296263 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 01:15:50.296759 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:50.296327 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 01:15:51.123822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.123779 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm"] Oct 30 01:15:51.139698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.139654 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:51.143002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.142973 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm"] Oct 30 01:15:51.249187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.249132 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 01:15:51.251177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.251099 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 01:15:51.310193 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.310152 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 01:15:51.310863 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.310826 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 01:15:51.320814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.320775 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:51.446962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.446854 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:51.562044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.562004 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:51.735739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:51.735624 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:52.248523 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:52.248474 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 01:15:52.248734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:52.248550 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 01:15:52.256840 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:52.256781 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:52.293414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:52.292570 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 01:15:52.293414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:52.292655 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 01:15:52.643736 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:52.613900 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:53.275172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:53.275125 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:53.278023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:53.276401 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:53.278023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:53.277692 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:15:53.297923 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:53.297889 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:53.298509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:53.298480 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:53.299318 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:53.299297 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:15:53.594676 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:53.594626 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:55.265906 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:55.265869 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:56.125379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.125321 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" podUID="be6f389c-edee-4033-ab14-1aec373be293" containerName="payload-processing" containerID="cri-o://d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da" gracePeriod=25 Oct 30 01:15:56.144037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.143985 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" podUID="b3136a9e-bba3-4a76-b3cf-4fc2d85eb025" containerName="payload-pre-processing" containerID="cri-o://cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac" gracePeriod=25 Oct 30 01:15:56.512003 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.511971 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:15:56.517071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.517032 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:15:56.641855 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.641818 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6f389c-edee-4033-ab14-1aec373be293-kube-api-access-k482d\" (UniqueName: \"kubernetes.io/projected/be6f389c-edee-4033-ab14-1aec373be293-kube-api-access-k482d\") pod \"be6f389c-edee-4033-ab14-1aec373be293\" (UID: \"be6f389c-edee-4033-ab14-1aec373be293\") " Oct 30 01:15:56.641855 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.641871 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6f389c-edee-4033-ab14-1aec373be293-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6f389c-edee-4033-ab14-1aec373be293-plugins-config-volume\") pod \"be6f389c-edee-4033-ab14-1aec373be293\" (UID: \"be6f389c-edee-4033-ab14-1aec373be293\") " Oct 30 01:15:56.642167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.642079 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-plugins-config-volume\") pod \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\" (UID: \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\") " Oct 30 01:15:56.642230 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.642167 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-kube-api-access-5dpfj\" (UniqueName: \"kubernetes.io/projected/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-kube-api-access-5dpfj\") pod \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\" (UID: \"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025\") " Oct 30 01:15:56.642419 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.642370 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6f389c-edee-4033-ab14-1aec373be293-plugins-config-volume" pod "be6f389c-edee-4033-ab14-1aec373be293" (UID: "be6f389c-edee-4033-ab14-1aec373be293"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:56.642813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.642695 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-plugins-config-volume" pod "b3136a9e-bba3-4a76-b3cf-4fc2d85eb025" (UID: "b3136a9e-bba3-4a76-b3cf-4fc2d85eb025"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:56.643031 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.643014 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:56.643116 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.643033 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6f389c-edee-4033-ab14-1aec373be293-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:56.644606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.644583 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6f389c-edee-4033-ab14-1aec373be293-kube-api-access-k482d" pod "be6f389c-edee-4033-ab14-1aec373be293" (UID: "be6f389c-edee-4033-ab14-1aec373be293"). InnerVolumeSpecName "kube-api-access-k482d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:56.645048 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.645024 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-kube-api-access-5dpfj" pod "b3136a9e-bba3-4a76-b3cf-4fc2d85eb025" (UID: "b3136a9e-bba3-4a76-b3cf-4fc2d85eb025"). InnerVolumeSpecName "kube-api-access-5dpfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:56.744611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.744513 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dpfj\" (UniqueName: \"kubernetes.io/projected/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025-kube-api-access-5dpfj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:56.744611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.744554 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k482d\" (UniqueName: \"kubernetes.io/projected/be6f389c-edee-4033-ab14-1aec373be293-kube-api-access-k482d\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:15:56.778033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.777988 4517 generic.go:356] "Generic (PLEG): container finished" podID="b3136a9e-bba3-4a76-b3cf-4fc2d85eb025" containerID="cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac" exitCode=0 Oct 30 01:15:56.778033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.778035 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" Oct 30 01:15:56.778353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.778030 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" event={"ID":"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025","Type":"ContainerDied","Data":"cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac"} Oct 30 01:15:56.778353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.778168 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm" event={"ID":"b3136a9e-bba3-4a76-b3cf-4fc2d85eb025","Type":"ContainerDied","Data":"c038457f863f970a9a127baa8284c78445316575fded2721671c6bd09979057b"} Oct 30 01:15:56.778353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.778201 4517 scope.go:122] "RemoveContainer" containerID="cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac" Oct 30 01:15:56.781162 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.781128 4517 generic.go:356] "Generic (PLEG): container finished" podID="be6f389c-edee-4033-ab14-1aec373be293" containerID="d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da" exitCode=0 Oct 30 01:15:56.781334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.781209 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" event={"ID":"be6f389c-edee-4033-ab14-1aec373be293","Type":"ContainerDied","Data":"d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da"} Oct 30 01:15:56.781334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.781265 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" event={"ID":"be6f389c-edee-4033-ab14-1aec373be293","Type":"ContainerDied","Data":"98f7b3df62023153ac5bec3a6a8574a19398a1865d65890ada0994e8aada96d9"} Oct 30 01:15:56.781457 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.781349 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm" Oct 30 01:15:56.794019 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.793993 4517 scope.go:122] "RemoveContainer" containerID="cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac" Oct 30 01:15:56.794445 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:56.794422 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac\": container with ID starting with cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac not found: ID does not exist" containerID="cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac" Oct 30 01:15:56.794525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.794453 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac"} err="failed to get container status \"cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac\": rpc error: code = NotFound desc = could not find container \"cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac\": container with ID starting with cd7eef26b2e9df3659860b2f4b8a5d621f5b309a5ef0d66f28d519fd110376ac not found: ID does not exist" Oct 30 01:15:56.794525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.794472 4517 scope.go:122] "RemoveContainer" containerID="d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da" Oct 30 01:15:56.808361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.808341 4517 scope.go:122] "RemoveContainer" containerID="d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da" Oct 30 01:15:56.808731 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:15:56.808691 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da\": container with ID starting with d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da not found: ID does not exist" containerID="d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da" Oct 30 01:15:56.808811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.808744 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da"} err="failed to get container status \"d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da\": rpc error: code = NotFound desc = could not find container \"d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da\": container with ID starting with d2edb9c174ee67de2386b8f6b9065351744902e9398bc4ef20caf6f9f3b3b1da not found: ID does not exist" Oct 30 01:15:56.827155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.827109 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm"] Oct 30 01:15:56.831815 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.831780 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-5dfdbbf4f5-jqjkm"] Oct 30 01:15:56.874900 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.874855 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm"] Oct 30 01:15:56.879527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.879479 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-65677df64-8bpkm"] Oct 30 01:15:56.971382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:56.971345 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:58.070946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:58.066886 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:58.243217 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:58.243177 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:58.244195 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:58.244166 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3136a9e-bba3-4a76-b3cf-4fc2d85eb025" path="/var/lib/kubelet/pods/b3136a9e-bba3-4a76-b3cf-4fc2d85eb025/volumes" Oct 30 01:15:58.244965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:58.244923 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6f389c-edee-4033-ab14-1aec373be293" path="/var/lib/kubelet/pods/be6f389c-edee-4033-ab14-1aec373be293/volumes" Oct 30 01:15:58.405767 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:58.405644 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:59.414799 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:15:59.414758 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:00.762604 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:16:00.762562 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:16:01.227363 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:01.227321 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:02.725577 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:16:02.725539 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:16:03.030234 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:03.030103 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:04.620259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:04.620208 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:06.443107 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:06.443068 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:07.251410 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:16:07.251368 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:16:08.108951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:08.108910 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:09.729362 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:09.729331 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:11.631430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:11.631392 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:13.837360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:13.837195 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:13.889471 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:13.889416 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.059751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.058848 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.088807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.082445 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.088807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.086895 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:16:14.088807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.086983 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.229737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.210812 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:14.277927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.277875 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.358305 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:16:14.358195 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:16:14.531873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.531813 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 01:16:14.599735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.595809 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.661625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.661498 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8"] Oct 30 01:16:14.663025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.662989 4517 cpu_manager.go:414] "containerMap: removing container" podUID="be6f389c-edee-4033-ab14-1aec373be293" containerName="payload-processing" Oct 30 01:16:14.663251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.663232 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6f389c-edee-4033-ab14-1aec373be293" containerName="payload-processing" Oct 30 01:16:14.663411 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.663396 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b3136a9e-bba3-4a76-b3cf-4fc2d85eb025" containerName="payload-pre-processing" Oct 30 01:16:14.663515 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.663501 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3136a9e-bba3-4a76-b3cf-4fc2d85eb025" containerName="payload-pre-processing" Oct 30 01:16:14.663868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.663848 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6f389c-edee-4033-ab14-1aec373be293" containerName="payload-processing" Oct 30 01:16:14.664106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.664090 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3136a9e-bba3-4a76-b3cf-4fc2d85eb025" containerName="payload-pre-processing" Oct 30 01:16:14.691750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.689304 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:14.730737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.724881 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1a42e5-openshift-default-dockercfg-n99zr\"" Oct 30 01:16:14.755915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.754085 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:14.778875 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.773621 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8"] Oct 30 01:16:14.843144 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.843091 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.908827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.844799 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 01:16:14.908827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.846280 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 01:16:14.908827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.850879 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.934798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.930850 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:16:14.934798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.931281 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 01:16:14.934798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.931856 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 01:16:15.020000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.998921 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istiod-ca-cert\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.020000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.999044 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-envoy\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.020000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.999168 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-token\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.020000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.999300 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-socket\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.020000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.999327 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-certs\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.020000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.999353 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-data\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.020000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.999439 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-credential-socket\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.020000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.999616 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-podinfo\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.020000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:14.999641 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7tw\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-kube-api-access-2g7tw\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.149849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.149795 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.161737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.150626 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 01:16:15.187748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.168485 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-podinfo\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.187748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.186792 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7tw\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-kube-api-access-2g7tw\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.187748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.186887 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istiod-ca-cert\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.187748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.187009 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-envoy\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.187748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.187158 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-token\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.187748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.187339 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-socket\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.187748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.187365 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-certs\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.187748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.187391 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-data\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.187748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.187476 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-credential-socket\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.220841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.220682 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istiod-ca-cert\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.228993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.221335 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-certs\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.228993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.221603 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-data\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.229894 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.229849 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.242047 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.230611 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 01:16:15.248039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.231055 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-socket\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.248303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.231257 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-credential-socket\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.248408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.239457 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:15.291134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.291094 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.291967 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.291944 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 01:16:15.357812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.337076 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 01:16:15.509170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.467963 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-envoy\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.509170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.494224 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.536739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.511823 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-token\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.538592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.537188 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 01:16:15.539104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.538459 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7tw\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-kube-api-access-2g7tw\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.567769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.540996 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 01:16:15.567769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.565618 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-podinfo\") pod \"e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.617763 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.613837 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.702739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.697186 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:15.752918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.752854 4517 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.810185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.784430 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 01:16:15.832184 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.832153 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 01:16:15.860091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.853045 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.860091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.857036 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 01:16:15.869222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.869188 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:15.918516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.918351 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 01:16:15.964058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.964012 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.966389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.964356 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 01:16:15.982598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:15.982570 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 01:16:16.008956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:16.008927 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 01:16:16.009974 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:16.009953 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 01:16:16.040512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:16.040478 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 01:16:16.526178 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:16:16.526133 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d57d9c_ceaa_4136_ab40_82d9e89e6cc2.slice/crio-58e8919c4517a43304e32e524a3cc7fceeb02b3f81e07d7fe123c58b53d38959 WatchSource:0}: Error finding container 58e8919c4517a43304e32e524a3cc7fceeb02b3f81e07d7fe123c58b53d38959: Status 404 returned error can't find the container with id 58e8919c4517a43304e32e524a3cc7fceeb02b3f81e07d7fe123c58b53d38959 Oct 30 01:16:16.527640 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:16.527598 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8"] Oct 30 01:16:16.529856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:16.529814 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:16:16.530002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:16.529896 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:16:16.530002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:16.529928 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:16:17.357737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:17.348880 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:17.399033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:17.398988 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" event={"ID":"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2","Type":"ContainerStarted","Data":"f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0"} Oct 30 01:16:17.399267 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:17.399067 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" event={"ID":"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2","Type":"ContainerStarted","Data":"58e8919c4517a43304e32e524a3cc7fceeb02b3f81e07d7fe123c58b53d38959"} Oct 30 01:16:17.457450 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:17.457369 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" podStartSLOduration=3.457341564 podStartE2EDuration="3.457341564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:17.450546582 +0000 UTC m=+2515.362673724" watchObservedRunningTime="2026-10-30 01:16:17.457341564 +0000 UTC m=+2515.369468709" Oct 30 01:16:17.769542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:17.769439 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:17.871424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:17.871378 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:17.879562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:17.879526 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:18.411785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:18.411692 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:18.413243 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:18.413212 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:16:18.933501 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:18.933465 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw"] Oct 30 01:16:18.938291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:18.938264 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:18.956508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:18.954803 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:16:18.956508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:18.954943 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:16:18.960626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:18.960589 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw"] Oct 30 01:16:19.024169 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.024134 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbj6\" (UniqueName: \"kubernetes.io/projected/92868a74-fbed-46a8-87c5-fc85ae88c7dd-kube-api-access-nvbj6\") pod \"maas-api-revoke-sub-7b98f505-v5kgw\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.024432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.024289 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b98f505-v5kgw\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.024517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.024476 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b98f505-v5kgw\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.125988 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.125950 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b98f505-v5kgw\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.126209 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.126058 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbj6\" (UniqueName: \"kubernetes.io/projected/92868a74-fbed-46a8-87c5-fc85ae88c7dd-kube-api-access-nvbj6\") pod \"maas-api-revoke-sub-7b98f505-v5kgw\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.126209 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.126173 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b98f505-v5kgw\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.126690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.126669 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b98f505-v5kgw\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.128821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.128796 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b98f505-v5kgw\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.148510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.148484 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbj6\" (UniqueName: \"kubernetes.io/projected/92868a74-fbed-46a8-87c5-fc85ae88c7dd-kube-api-access-nvbj6\") pod \"maas-api-revoke-sub-7b98f505-v5kgw\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.253361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.253264 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:19.360044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.360004 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:19.563281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:19.563254 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw"] Oct 30 01:16:19.565121 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:16:19.565076 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92868a74_fbed_46a8_87c5_fc85ae88c7dd.slice/crio-3c0f2067d85f671ca94ffd20135d0c2e6a262206cf429faa93c5a6e5866081cb WatchSource:0}: Error finding container 3c0f2067d85f671ca94ffd20135d0c2e6a262206cf429faa93c5a6e5866081cb: Status 404 returned error can't find the container with id 3c0f2067d85f671ca94ffd20135d0c2e6a262206cf429faa93c5a6e5866081cb Oct 30 01:16:20.000945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:20.000907 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:20.126221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:20.126130 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:20.504841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:20.504690 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" event={"ID":"92868a74-fbed-46a8-87c5-fc85ae88c7dd","Type":"ContainerStarted","Data":"10b9f41b7dac3742a1a08b810d2d5ca341742aeda01f2c96f20c14a32332728b"} Oct 30 01:16:20.504841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:20.504801 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" event={"ID":"92868a74-fbed-46a8-87c5-fc85ae88c7dd","Type":"ContainerStarted","Data":"3c0f2067d85f671ca94ffd20135d0c2e6a262206cf429faa93c5a6e5866081cb"} Oct 30 01:16:20.545923 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:20.545866 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" podStartSLOduration=2.545851959 podStartE2EDuration="2.545851959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:20.542370277 +0000 UTC m=+2518.454497418" watchObservedRunningTime="2026-10-30 01:16:20.545851959 +0000 UTC m=+2518.457979099" Oct 30 01:16:21.460285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:21.460237 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:21.540499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:21.540457 4517 generic.go:356] "Generic (PLEG): container finished" podID="92868a74-fbed-46a8-87c5-fc85ae88c7dd" containerID="10b9f41b7dac3742a1a08b810d2d5ca341742aeda01f2c96f20c14a32332728b" exitCode=0 Oct 30 01:16:21.540936 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:21.540901 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" event={"ID":"92868a74-fbed-46a8-87c5-fc85ae88c7dd","Type":"ContainerDied","Data":"10b9f41b7dac3742a1a08b810d2d5ca341742aeda01f2c96f20c14a32332728b"} Oct 30 01:16:21.829817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:21.829779 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:21.946927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:21.946888 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:22.004103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.004064 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7"] Oct 30 01:16:22.008606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.008579 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:22.024647 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.024619 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a42e5\"" Oct 30 01:16:22.032116 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.032085 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a42e5-dockercfg-khx2x\"" Oct 30 01:16:22.036493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.036461 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7"] Oct 30 01:16:22.217044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.214515 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7\" (UID: \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:22.217044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.214695 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjmf\" (UniqueName: \"kubernetes.io/projected/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-kube-api-access-fhjmf\") pod \"payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7\" (UID: \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:22.375284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.374495 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7\" (UID: \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:22.375284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.374649 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjmf\" (UniqueName: \"kubernetes.io/projected/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-kube-api-access-fhjmf\") pod \"payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7\" (UID: \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:22.479159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.472089 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a42e5\"" Oct 30 01:16:22.479159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.478882 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7\" (UID: \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:22.527628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.520485 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjmf\" (UniqueName: \"kubernetes.io/projected/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-kube-api-access-fhjmf\") pod \"payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7\" (UID: \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:22.585051 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:16:22.585012 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f77bcc_c251_49e5_a309_9dede3b87810.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:16:22.636048 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.636010 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a42e5-dockercfg-khx2x\"" Oct 30 01:16:22.644647 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.644603 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:22.769073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.769038 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:22.782158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.782118 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92868a74-fbed-46a8-87c5-fc85ae88c7dd-kube-api-access-nvbj6\" (UniqueName: \"kubernetes.io/projected/92868a74-fbed-46a8-87c5-fc85ae88c7dd-kube-api-access-nvbj6\") pod \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " Oct 30 01:16:22.782369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.782257 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-openshift-ca\") pod \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " Oct 30 01:16:22.782442 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.782371 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-xks-ca\") pod \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\" (UID: \"92868a74-fbed-46a8-87c5-fc85ae88c7dd\") " Oct 30 01:16:22.782735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.782667 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-openshift-ca" pod "92868a74-fbed-46a8-87c5-fc85ae88c7dd" (UID: "92868a74-fbed-46a8-87c5-fc85ae88c7dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:22.783218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.783186 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:22.785278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.785248 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92868a74-fbed-46a8-87c5-fc85ae88c7dd-kube-api-access-nvbj6" pod "92868a74-fbed-46a8-87c5-fc85ae88c7dd" (UID: "92868a74-fbed-46a8-87c5-fc85ae88c7dd"). InnerVolumeSpecName "kube-api-access-nvbj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:22.786180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.786156 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-xks-ca" pod "92868a74-fbed-46a8-87c5-fc85ae88c7dd" (UID: "92868a74-fbed-46a8-87c5-fc85ae88c7dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:16:22.885135 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.885090 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbj6\" (UniqueName: \"kubernetes.io/projected/92868a74-fbed-46a8-87c5-fc85ae88c7dd-kube-api-access-nvbj6\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:22.885135 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.885128 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92868a74-fbed-46a8-87c5-fc85ae88c7dd-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:22.990543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:22.990509 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7"] Oct 30 01:16:22.992694 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:16:22.992663 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea1f145_1eb3_49a4_8658_b32e7b228ebb.slice/crio-3f699f40baaf3e529edb90c20e9384228acb9236a5cc21e8cbccfe04b797890f WatchSource:0}: Error finding container 3f699f40baaf3e529edb90c20e9384228acb9236a5cc21e8cbccfe04b797890f: Status 404 returned error can't find the container with id 3f699f40baaf3e529edb90c20e9384228acb9236a5cc21e8cbccfe04b797890f Oct 30 01:16:23.524063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:23.524030 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:23.633298 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:23.633255 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" Oct 30 01:16:23.633298 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:23.633257 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw" event={"ID":"92868a74-fbed-46a8-87c5-fc85ae88c7dd","Type":"ContainerDied","Data":"3c0f2067d85f671ca94ffd20135d0c2e6a262206cf429faa93c5a6e5866081cb"} Oct 30 01:16:23.633792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:23.633411 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c0f2067d85f671ca94ffd20135d0c2e6a262206cf429faa93c5a6e5866081cb" Oct 30 01:16:23.637240 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:23.636221 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" event={"ID":"4ea1f145-1eb3-49a4-8658-b32e7b228ebb","Type":"ContainerStarted","Data":"4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674"} Oct 30 01:16:23.637240 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:23.636299 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" event={"ID":"4ea1f145-1eb3-49a4-8658-b32e7b228ebb","Type":"ContainerStarted","Data":"3f699f40baaf3e529edb90c20e9384228acb9236a5cc21e8cbccfe04b797890f"} Oct 30 01:16:23.637802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:23.637764 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:25.313615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:25.308278 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:27.077052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:27.077018 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:27.813274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:27.813224 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:29.062070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.062019 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:29.335291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.335142 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" podStartSLOduration=8.335113881 podStartE2EDuration="8.335113881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:23.681939676 +0000 UTC m=+2521.594066817" watchObservedRunningTime="2026-10-30 01:16:29.335113881 +0000 UTC m=+2527.247241020" Oct 30 01:16:29.339095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.339067 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k"] Oct 30 01:16:29.339840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.339824 4517 cpu_manager.go:414] "containerMap: removing container" podUID="92868a74-fbed-46a8-87c5-fc85ae88c7dd" containerName="revoke-keys" Oct 30 01:16:29.339914 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.339841 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92868a74-fbed-46a8-87c5-fc85ae88c7dd" containerName="revoke-keys" Oct 30 01:16:29.339986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.339977 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="92868a74-fbed-46a8-87c5-fc85ae88c7dd" containerName="revoke-keys" Oct 30 01:16:29.345118 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.345097 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.364757 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.364726 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k"] Oct 30 01:16:29.367789 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.367765 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:16:29.368513 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.368491 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:16:29.437860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.437821 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rf5x\" (UniqueName: \"kubernetes.io/projected/7adb12b2-39c9-474d-aafa-621b23281665-kube-api-access-6rf5x\") pod \"maas-api-revoke-sub-61ad374b-w2n5k\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.438040 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.437916 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61ad374b-w2n5k\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.438040 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.437998 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61ad374b-w2n5k\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.539411 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.539377 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61ad374b-w2n5k\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.539627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.539611 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rf5x\" (UniqueName: \"kubernetes.io/projected/7adb12b2-39c9-474d-aafa-621b23281665-kube-api-access-6rf5x\") pod \"maas-api-revoke-sub-61ad374b-w2n5k\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.539678 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.539661 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61ad374b-w2n5k\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.540163 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.540139 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61ad374b-w2n5k\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.542209 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.542181 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61ad374b-w2n5k\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.562022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.561992 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rf5x\" (UniqueName: \"kubernetes.io/projected/7adb12b2-39c9-474d-aafa-621b23281665-kube-api-access-6rf5x\") pod \"maas-api-revoke-sub-61ad374b-w2n5k\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.662000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.661904 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:29.983739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:29.983693 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k"] Oct 30 01:16:29.985322 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:16:29.985288 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adb12b2_39c9_474d_aafa_621b23281665.slice/crio-661d671a634083dc2d9fe909725f09fdfd56233d56c785747838d2fc4baf8a77 WatchSource:0}: Error finding container 661d671a634083dc2d9fe909725f09fdfd56233d56c785747838d2fc4baf8a77: Status 404 returned error can't find the container with id 661d671a634083dc2d9fe909725f09fdfd56233d56c785747838d2fc4baf8a77 Oct 30 01:16:30.698913 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:30.698872 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" event={"ID":"7adb12b2-39c9-474d-aafa-621b23281665","Type":"ContainerStarted","Data":"935f30db20bf788e127783881a091978b19bfc8cead2e7485a6d43be39cc7038"} Oct 30 01:16:30.699454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:30.698934 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" event={"ID":"7adb12b2-39c9-474d-aafa-621b23281665","Type":"ContainerStarted","Data":"661d671a634083dc2d9fe909725f09fdfd56233d56c785747838d2fc4baf8a77"} Oct 30 01:16:30.739218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:30.739166 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" podStartSLOduration=1.739150637 podStartE2EDuration="1.739150637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:30.736355508 +0000 UTC m=+2528.648482649" watchObservedRunningTime="2026-10-30 01:16:30.739150637 +0000 UTC m=+2528.651277775" Oct 30 01:16:30.763753 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:16:30.763677 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:16:30.903810 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:30.903776 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:31.708854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:31.708809 4517 generic.go:356] "Generic (PLEG): container finished" podID="7adb12b2-39c9-474d-aafa-621b23281665" containerID="935f30db20bf788e127783881a091978b19bfc8cead2e7485a6d43be39cc7038" exitCode=0 Oct 30 01:16:31.709279 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:31.708906 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" event={"ID":"7adb12b2-39c9-474d-aafa-621b23281665","Type":"ContainerDied","Data":"935f30db20bf788e127783881a091978b19bfc8cead2e7485a6d43be39cc7038"} Oct 30 01:16:32.756348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:32.756247 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:32.968405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:32.968369 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:32.996774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:32.996678 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adb12b2-39c9-474d-aafa-621b23281665-kube-api-access-6rf5x\" (UniqueName: \"kubernetes.io/projected/7adb12b2-39c9-474d-aafa-621b23281665-kube-api-access-6rf5x\") pod \"7adb12b2-39c9-474d-aafa-621b23281665\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " Oct 30 01:16:32.996952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:32.996874 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-xks-ca\") pod \"7adb12b2-39c9-474d-aafa-621b23281665\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " Oct 30 01:16:32.996952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:32.996945 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-openshift-ca\") pod \"7adb12b2-39c9-474d-aafa-621b23281665\" (UID: \"7adb12b2-39c9-474d-aafa-621b23281665\") " Oct 30 01:16:32.997463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:32.997369 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-openshift-ca" pod "7adb12b2-39c9-474d-aafa-621b23281665" (UID: "7adb12b2-39c9-474d-aafa-621b23281665"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:32.997740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:32.997699 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:32.999369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:32.999345 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adb12b2-39c9-474d-aafa-621b23281665-kube-api-access-6rf5x" pod "7adb12b2-39c9-474d-aafa-621b23281665" (UID: "7adb12b2-39c9-474d-aafa-621b23281665"). InnerVolumeSpecName "kube-api-access-6rf5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:32.999658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:32.999635 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-xks-ca" pod "7adb12b2-39c9-474d-aafa-621b23281665" (UID: "7adb12b2-39c9-474d-aafa-621b23281665"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:16:33.099197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:33.099159 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rf5x\" (UniqueName: \"kubernetes.io/projected/7adb12b2-39c9-474d-aafa-621b23281665-kube-api-access-6rf5x\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:33.099197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:33.099193 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adb12b2-39c9-474d-aafa-621b23281665-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:33.738332 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:33.738299 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" Oct 30 01:16:33.738546 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:33.738335 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k" event={"ID":"7adb12b2-39c9-474d-aafa-621b23281665","Type":"ContainerDied","Data":"661d671a634083dc2d9fe909725f09fdfd56233d56c785747838d2fc4baf8a77"} Oct 30 01:16:33.738546 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:33.738380 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="661d671a634083dc2d9fe909725f09fdfd56233d56c785747838d2fc4baf8a77" Oct 30 01:16:34.617742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:34.603355 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:34.685139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:34.678449 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:16:35.189318 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.189281 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7"] Oct 30 01:16:35.190051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.190036 4517 cpu_manager.go:414] "containerMap: removing container" podUID="7adb12b2-39c9-474d-aafa-621b23281665" containerName="revoke-keys" Oct 30 01:16:35.190101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.190053 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adb12b2-39c9-474d-aafa-621b23281665" containerName="revoke-keys" Oct 30 01:16:35.190163 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.190154 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adb12b2-39c9-474d-aafa-621b23281665" containerName="revoke-keys" Oct 30 01:16:35.193924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.193905 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.209364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.209330 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f69104-openshift-default-dockercfg-2b6hn\"" Oct 30 01:16:35.223378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.223338 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7"] Oct 30 01:16:35.236569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.236532 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istiod-ca-cert\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.236922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.236895 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-certs\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.238968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.238934 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-credential-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.239081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.238993 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-token\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.239081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.239050 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-envoy\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.239207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.239095 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-podinfo\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.239270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.239217 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-data\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.239321 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.239265 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.239429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.239409 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq7f2\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-kube-api-access-cq7f2\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.340538 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.340501 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-envoy\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.340772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.340553 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-podinfo\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.340924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.340803 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-data\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.340924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.340869 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.341122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.341085 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7f2\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-kube-api-access-cq7f2\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.341262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.341229 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.341341 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.341282 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-data\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.341341 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.341291 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istiod-ca-cert\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.341472 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.341440 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-certs\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.341536 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.341493 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-credential-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.341602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.341532 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-token\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.342002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.341928 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istiod-ca-cert\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.342114 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.342042 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-certs\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.342174 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.342124 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-credential-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.343608 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.343581 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-podinfo\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.344189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.344158 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-envoy\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.368131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.368094 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7f2\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-kube-api-access-cq7f2\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.368281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.368225 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-token\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.511774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.511637 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:35.845194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.845161 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7"] Oct 30 01:16:35.846455 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:16:35.846426 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e39131_678d_4eb0_9c12_f9c1e768c3bb.slice/crio-4c71eb0eb7b73ab2fce0b58448f822c3c54483129d18a91fca296a451dec4c16 WatchSource:0}: Error finding container 4c71eb0eb7b73ab2fce0b58448f822c3c54483129d18a91fca296a451dec4c16: Status 404 returned error can't find the container with id 4c71eb0eb7b73ab2fce0b58448f822c3c54483129d18a91fca296a451dec4c16 Oct 30 01:16:35.849593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.849552 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:16:35.849760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.849630 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:16:35.849760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:35.849661 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:16:36.499260 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:36.499221 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:36.764740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:36.764619 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" event={"ID":"28e39131-678d-4eb0-9c12-f9c1e768c3bb","Type":"ContainerStarted","Data":"d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5"} Oct 30 01:16:36.764740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:36.764677 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" event={"ID":"28e39131-678d-4eb0-9c12-f9c1e768c3bb","Type":"ContainerStarted","Data":"4c71eb0eb7b73ab2fce0b58448f822c3c54483129d18a91fca296a451dec4c16"} Oct 30 01:16:36.823361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:36.823294 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" podStartSLOduration=1.823272357 podStartE2EDuration="1.823272357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:36.818260912 +0000 UTC m=+2534.730388053" watchObservedRunningTime="2026-10-30 01:16:36.823272357 +0000 UTC m=+2534.735399502" Oct 30 01:16:37.512132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:37.512097 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:37.513739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:37.513681 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 01:16:37.513873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:37.513765 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 01:16:38.153020 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:38.151417 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf"] Oct 30 01:16:38.192455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:38.192418 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:38.193898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:38.193871 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869"] Oct 30 01:16:38.327894 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:38.327846 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:38.537278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:38.537241 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 01:16:38.551237 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:38.551184 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 01:16:38.552084 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:38.539319 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:38.702062 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:38.702027 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:39.513968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:39.513933 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 01:16:39.514326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:39.514291 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 01:16:39.599300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:39.599267 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:39.843679 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:39.843623 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:40.533108 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.533074 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 01:16:40.545729 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.533860 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 01:16:40.554430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.550054 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww"] Oct 30 01:16:40.572086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.572052 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.596877 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.596821 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a1de2e-cwqww\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.597282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.597258 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a1de2e-cwqww\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.597581 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.597559 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmvk\" (UniqueName: \"kubernetes.io/projected/65ea793b-b7d2-4fad-8347-f73c4667467c-kube-api-access-6gmvk\") pod \"maas-api-revoke-sub-06a1de2e-cwqww\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.597920 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.597903 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww"] Oct 30 01:16:40.655739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.655687 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:16:40.657587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.656633 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:16:40.702422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.702377 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a1de2e-cwqww\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.702610 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.702528 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a1de2e-cwqww\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.702694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.702670 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmvk\" (UniqueName: \"kubernetes.io/projected/65ea793b-b7d2-4fad-8347-f73c4667467c-kube-api-access-6gmvk\") pod \"maas-api-revoke-sub-06a1de2e-cwqww\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.703298 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.703266 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a1de2e-cwqww\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.706133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.706109 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a1de2e-cwqww\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.728066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.728035 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmvk\" (UniqueName: \"kubernetes.io/projected/65ea793b-b7d2-4fad-8347-f73c4667467c-kube-api-access-6gmvk\") pod \"maas-api-revoke-sub-06a1de2e-cwqww\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.954093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.954047 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:40.988732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:40.983382 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:41.500366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:41.499428 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww"] Oct 30 01:16:41.502951 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:16:41.502917 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ea793b_b7d2_4fad_8347_f73c4667467c.slice/crio-e501a2d27159bcec717fdd417e57a84944c4016786593aa08ae3129990c58376 WatchSource:0}: Error finding container e501a2d27159bcec717fdd417e57a84944c4016786593aa08ae3129990c58376: Status 404 returned error can't find the container with id e501a2d27159bcec717fdd417e57a84944c4016786593aa08ae3129990c58376 Oct 30 01:16:41.512661 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:41.512618 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 01:16:41.512840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:41.512695 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 01:16:42.192735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:42.191902 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" event={"ID":"65ea793b-b7d2-4fad-8347-f73c4667467c","Type":"ContainerStarted","Data":"9354ac7af7cb5b538e54fc9e09383daa76abaa03412f100ffbfba621947bfd95"} Oct 30 01:16:42.192735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:42.191987 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" event={"ID":"65ea793b-b7d2-4fad-8347-f73c4667467c","Type":"ContainerStarted","Data":"e501a2d27159bcec717fdd417e57a84944c4016786593aa08ae3129990c58376"} Oct 30 01:16:42.231064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:42.230988 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" podStartSLOduration=2.230969245 podStartE2EDuration="2.230969245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:42.229268984 +0000 UTC m=+2540.141396124" watchObservedRunningTime="2026-10-30 01:16:42.230969245 +0000 UTC m=+2540.143096388" Oct 30 01:16:42.513009 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:42.512972 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 01:16:42.513226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:42.513028 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 01:16:42.849150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:42.849113 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:43.152571 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.152459 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" podUID="6ebe5e3d-0433-402b-82cd-19af9889bd6e" containerName="payload-processing" containerID="cri-o://c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7" gracePeriod=25 Oct 30 01:16:43.195570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.194913 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" podUID="8093216b-6714-405c-ab9d-d02d47e7de25" containerName="payload-pre-processing" containerID="cri-o://027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99" gracePeriod=25 Oct 30 01:16:43.208414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.208373 4517 generic.go:356] "Generic (PLEG): container finished" podID="65ea793b-b7d2-4fad-8347-f73c4667467c" containerID="9354ac7af7cb5b538e54fc9e09383daa76abaa03412f100ffbfba621947bfd95" exitCode=0 Oct 30 01:16:43.208594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.208471 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" event={"ID":"65ea793b-b7d2-4fad-8347-f73c4667467c","Type":"ContainerDied","Data":"9354ac7af7cb5b538e54fc9e09383daa76abaa03412f100ffbfba621947bfd95"} Oct 30 01:16:43.513007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.512968 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 01:16:43.513185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.513027 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 01:16:43.586785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.586752 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:16:43.618100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.618071 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:16:43.733452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.733340 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8093216b-6714-405c-ab9d-d02d47e7de25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093216b-6714-405c-ab9d-d02d47e7de25-plugins-config-volume\") pod \"8093216b-6714-405c-ab9d-d02d47e7de25\" (UID: \"8093216b-6714-405c-ab9d-d02d47e7de25\") " Oct 30 01:16:43.733452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.733413 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8093216b-6714-405c-ab9d-d02d47e7de25-kube-api-access-2td2b\" (UniqueName: \"kubernetes.io/projected/8093216b-6714-405c-ab9d-d02d47e7de25-kube-api-access-2td2b\") pod \"8093216b-6714-405c-ab9d-d02d47e7de25\" (UID: \"8093216b-6714-405c-ab9d-d02d47e7de25\") " Oct 30 01:16:43.733756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.733504 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebe5e3d-0433-402b-82cd-19af9889bd6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebe5e3d-0433-402b-82cd-19af9889bd6e-plugins-config-volume\") pod \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\" (UID: \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\") " Oct 30 01:16:43.733879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.733843 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8093216b-6714-405c-ab9d-d02d47e7de25-plugins-config-volume" pod "8093216b-6714-405c-ab9d-d02d47e7de25" (UID: "8093216b-6714-405c-ab9d-d02d47e7de25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:43.733976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.733954 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebe5e3d-0433-402b-82cd-19af9889bd6e-kube-api-access-tcww6\" (UniqueName: \"kubernetes.io/projected/6ebe5e3d-0433-402b-82cd-19af9889bd6e-kube-api-access-tcww6\") pod \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\" (UID: \"6ebe5e3d-0433-402b-82cd-19af9889bd6e\") " Oct 30 01:16:43.734272 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.734241 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebe5e3d-0433-402b-82cd-19af9889bd6e-plugins-config-volume" pod "6ebe5e3d-0433-402b-82cd-19af9889bd6e" (UID: "6ebe5e3d-0433-402b-82cd-19af9889bd6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:43.735236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.735203 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093216b-6714-405c-ab9d-d02d47e7de25-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:43.735366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.735238 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebe5e3d-0433-402b-82cd-19af9889bd6e-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:43.736567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.736522 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8093216b-6714-405c-ab9d-d02d47e7de25-kube-api-access-2td2b" pod "8093216b-6714-405c-ab9d-d02d47e7de25" (UID: "8093216b-6714-405c-ab9d-d02d47e7de25"). InnerVolumeSpecName "kube-api-access-2td2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:43.736992 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.736966 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebe5e3d-0433-402b-82cd-19af9889bd6e-kube-api-access-tcww6" pod "6ebe5e3d-0433-402b-82cd-19af9889bd6e" (UID: "6ebe5e3d-0433-402b-82cd-19af9889bd6e"). InnerVolumeSpecName "kube-api-access-tcww6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:43.836721 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.836653 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2td2b\" (UniqueName: \"kubernetes.io/projected/8093216b-6714-405c-ab9d-d02d47e7de25-kube-api-access-2td2b\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:43.836721 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:43.836697 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcww6\" (UniqueName: \"kubernetes.io/projected/6ebe5e3d-0433-402b-82cd-19af9889bd6e-kube-api-access-tcww6\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:44.310737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.307946 4517 generic.go:356] "Generic (PLEG): container finished" podID="8093216b-6714-405c-ab9d-d02d47e7de25" containerID="027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99" exitCode=0 Oct 30 01:16:44.310737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.308152 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" Oct 30 01:16:44.337736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.330603 4517 generic.go:356] "Generic (PLEG): container finished" podID="6ebe5e3d-0433-402b-82cd-19af9889bd6e" containerID="c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7" exitCode=0 Oct 30 01:16:44.337736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.330965 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" Oct 30 01:16:44.447285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.445637 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" event={"ID":"8093216b-6714-405c-ab9d-d02d47e7de25","Type":"ContainerDied","Data":"027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99"} Oct 30 01:16:44.447285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.445741 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869" event={"ID":"8093216b-6714-405c-ab9d-d02d47e7de25","Type":"ContainerDied","Data":"a5b54a3916588b865e85557c7b8044b8ce4df037ad6c26e63c26c8a082ed1ccf"} Oct 30 01:16:44.447285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.445773 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" event={"ID":"6ebe5e3d-0433-402b-82cd-19af9889bd6e","Type":"ContainerDied","Data":"c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7"} Oct 30 01:16:44.447285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.445802 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf" event={"ID":"6ebe5e3d-0433-402b-82cd-19af9889bd6e","Type":"ContainerDied","Data":"757ebf4caea581cd770b7dd79d8cbcd7aed2ef4bb31479461e7b5bb8678f7304"} Oct 30 01:16:44.447285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.445833 4517 scope.go:122] "RemoveContainer" containerID="027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99" Oct 30 01:16:44.550246 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.550175 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:44.552226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.552105 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:44.584480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.584444 4517 scope.go:122] "RemoveContainer" containerID="027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99" Oct 30 01:16:44.584695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.584639 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:16:44.604589 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:16:44.604550 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99\": container with ID starting with 027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99 not found: ID does not exist" containerID="027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99" Oct 30 01:16:44.604870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.604833 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99"} err="failed to get container status \"027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99\": rpc error: code = NotFound desc = could not find container \"027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99\": container with ID starting with 027ae081448399f731643cc9d7b9e55d997fd6ea38faf7b0ed84a9f345214c99 not found: ID does not exist" Oct 30 01:16:44.605026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.605012 4517 scope.go:122] "RemoveContainer" containerID="c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7" Oct 30 01:16:44.634293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.634203 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf"] Oct 30 01:16:44.712175 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.710931 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-5b7f5777f8-fslvf"] Oct 30 01:16:44.835147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.813921 4517 scope.go:122] "RemoveContainer" containerID="c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7" Oct 30 01:16:44.919432 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:16:44.895277 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7\": container with ID starting with c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7 not found: ID does not exist" containerID="c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7" Oct 30 01:16:44.919432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.895327 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7"} err="failed to get container status \"c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7\": rpc error: code = NotFound desc = could not find container \"c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7\": container with ID starting with c2bcb4ae748f1b30cb76f776dc52b55a8caef49131e83062545ffe8053a40ac7 not found: ID does not exist" Oct 30 01:16:44.982733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:44.973498 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869"] Oct 30 01:16:45.042562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.042018 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7f58d987b7-9h869"] Oct 30 01:16:45.138415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.138378 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:45.174588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.174554 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:45.334272 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.334237 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-openshift-ca\") pod \"65ea793b-b7d2-4fad-8347-f73c4667467c\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " Oct 30 01:16:45.335003 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.334977 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ea793b-b7d2-4fad-8347-f73c4667467c-kube-api-access-6gmvk\" (UniqueName: \"kubernetes.io/projected/65ea793b-b7d2-4fad-8347-f73c4667467c-kube-api-access-6gmvk\") pod \"65ea793b-b7d2-4fad-8347-f73c4667467c\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " Oct 30 01:16:45.335365 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.335346 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-xks-ca\") pod \"65ea793b-b7d2-4fad-8347-f73c4667467c\" (UID: \"65ea793b-b7d2-4fad-8347-f73c4667467c\") " Oct 30 01:16:45.337288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.337252 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-openshift-ca" pod "65ea793b-b7d2-4fad-8347-f73c4667467c" (UID: "65ea793b-b7d2-4fad-8347-f73c4667467c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:45.343489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.343365 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ea793b-b7d2-4fad-8347-f73c4667467c-kube-api-access-6gmvk" pod "65ea793b-b7d2-4fad-8347-f73c4667467c" (UID: "65ea793b-b7d2-4fad-8347-f73c4667467c"). InnerVolumeSpecName "kube-api-access-6gmvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:45.345508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.345476 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" event={"ID":"65ea793b-b7d2-4fad-8347-f73c4667467c","Type":"ContainerDied","Data":"e501a2d27159bcec717fdd417e57a84944c4016786593aa08ae3129990c58376"} Oct 30 01:16:45.345724 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.345537 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e501a2d27159bcec717fdd417e57a84944c4016786593aa08ae3129990c58376" Oct 30 01:16:45.345724 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.345615 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww" Oct 30 01:16:45.349363 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.349331 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-xks-ca" pod "65ea793b-b7d2-4fad-8347-f73c4667467c" (UID: "65ea793b-b7d2-4fad-8347-f73c4667467c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:16:45.439324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.439233 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:45.439324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.439267 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ea793b-b7d2-4fad-8347-f73c4667467c-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:45.439324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:45.439282 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gmvk\" (UniqueName: \"kubernetes.io/projected/65ea793b-b7d2-4fad-8347-f73c4667467c-kube-api-access-6gmvk\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:46.237604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:46.237559 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebe5e3d-0433-402b-82cd-19af9889bd6e" path="/var/lib/kubelet/pods/6ebe5e3d-0433-402b-82cd-19af9889bd6e/volumes" Oct 30 01:16:46.238185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:46.238167 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8093216b-6714-405c-ab9d-d02d47e7de25" path="/var/lib/kubelet/pods/8093216b-6714-405c-ab9d-d02d47e7de25/volumes" Oct 30 01:16:47.808736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:47.807488 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:48.767642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:48.767594 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5"] Oct 30 01:16:48.889421 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:48.887390 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:49.039792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:49.039310 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:49.989209 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:49.989168 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:50.961302 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.961256 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk"] Oct 30 01:16:50.962379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.962352 4517 cpu_manager.go:414] "containerMap: removing container" podUID="6ebe5e3d-0433-402b-82cd-19af9889bd6e" containerName="payload-processing" Oct 30 01:16:50.962521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.962382 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebe5e3d-0433-402b-82cd-19af9889bd6e" containerName="payload-processing" Oct 30 01:16:50.962521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.962425 4517 cpu_manager.go:414] "containerMap: removing container" podUID="8093216b-6714-405c-ab9d-d02d47e7de25" containerName="payload-pre-processing" Oct 30 01:16:50.962521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.962433 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8093216b-6714-405c-ab9d-d02d47e7de25" containerName="payload-pre-processing" Oct 30 01:16:50.962521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.962491 4517 cpu_manager.go:414] "containerMap: removing container" podUID="65ea793b-b7d2-4fad-8347-f73c4667467c" containerName="revoke-keys" Oct 30 01:16:50.962521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.962499 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ea793b-b7d2-4fad-8347-f73c4667467c" containerName="revoke-keys" Oct 30 01:16:50.962752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.962647 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="8093216b-6714-405c-ab9d-d02d47e7de25" containerName="payload-pre-processing" Oct 30 01:16:50.962752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.962671 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ea793b-b7d2-4fad-8347-f73c4667467c" containerName="revoke-keys" Oct 30 01:16:50.962752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.962685 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebe5e3d-0433-402b-82cd-19af9889bd6e" containerName="payload-processing" Oct 30 01:16:50.967307 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.967278 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:50.986022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.985988 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk"] Oct 30 01:16:50.988790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.988756 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:16:50.988976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:50.988831 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:16:51.075350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.075306 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-630db937-kcbtk\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.075867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.075451 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgqs\" (UniqueName: \"kubernetes.io/projected/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-kube-api-access-flgqs\") pod \"maas-api-revoke-sub-630db937-kcbtk\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.075867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.075598 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-630db937-kcbtk\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.177403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.177346 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-630db937-kcbtk\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.177677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.177427 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgqs\" (UniqueName: \"kubernetes.io/projected/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-kube-api-access-flgqs\") pod \"maas-api-revoke-sub-630db937-kcbtk\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.177677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.177529 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-630db937-kcbtk\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.178275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.178246 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-630db937-kcbtk\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.180575 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.180535 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-630db937-kcbtk\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.200576 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.200526 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgqs\" (UniqueName: \"kubernetes.io/projected/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-kube-api-access-flgqs\") pod \"maas-api-revoke-sub-630db937-kcbtk\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.286066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.286029 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:51.612890 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:51.612853 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk"] Oct 30 01:16:51.615697 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:16:51.615659 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3ce1c0_ca57_48b0_a758_08177f5f0af5.slice/crio-6f4d8d3f532d014abfdd5ba20fab82c9b7829803d3fadad8c6ad7d40f3bc912d WatchSource:0}: Error finding container 6f4d8d3f532d014abfdd5ba20fab82c9b7829803d3fadad8c6ad7d40f3bc912d: Status 404 returned error can't find the container with id 6f4d8d3f532d014abfdd5ba20fab82c9b7829803d3fadad8c6ad7d40f3bc912d Oct 30 01:16:52.547980 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:52.547940 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:52.547980 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:52.547944 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" event={"ID":"7c3ce1c0-ca57-48b0-a758-08177f5f0af5","Type":"ContainerStarted","Data":"71e87236eb868f0a659e4ad77a86bd7ca9bc4e0e6e96050bcb52894dd080940c"} Oct 30 01:16:52.548665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:52.548034 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" event={"ID":"7c3ce1c0-ca57-48b0-a758-08177f5f0af5","Type":"ContainerStarted","Data":"6f4d8d3f532d014abfdd5ba20fab82c9b7829803d3fadad8c6ad7d40f3bc912d"} Oct 30 01:16:53.559134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:53.559087 4517 generic.go:356] "Generic (PLEG): container finished" podID="7c3ce1c0-ca57-48b0-a758-08177f5f0af5" containerID="71e87236eb868f0a659e4ad77a86bd7ca9bc4e0e6e96050bcb52894dd080940c" exitCode=0 Oct 30 01:16:53.559670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:53.559182 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" event={"ID":"7c3ce1c0-ca57-48b0-a758-08177f5f0af5","Type":"ContainerDied","Data":"71e87236eb868f0a659e4ad77a86bd7ca9bc4e0e6e96050bcb52894dd080940c"} Oct 30 01:16:53.768892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:53.768793 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" podUID="ef6bf45e-8681-41e5-8d27-ebe94785eee7" containerName="payload-pre-processing" containerID="cri-o://4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a" gracePeriod=25 Oct 30 01:16:54.105833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.105793 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:16:54.167571 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.167456 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6bf45e-8681-41e5-8d27-ebe94785eee7-kube-api-access-5k8df\" (UniqueName: \"kubernetes.io/projected/ef6bf45e-8681-41e5-8d27-ebe94785eee7-kube-api-access-5k8df\") pod \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\" (UID: \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\") " Oct 30 01:16:54.167802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.167639 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6bf45e-8681-41e5-8d27-ebe94785eee7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6bf45e-8681-41e5-8d27-ebe94785eee7-plugins-config-volume\") pod \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\" (UID: \"ef6bf45e-8681-41e5-8d27-ebe94785eee7\") " Oct 30 01:16:54.169431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.169381 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6bf45e-8681-41e5-8d27-ebe94785eee7-plugins-config-volume" pod "ef6bf45e-8681-41e5-8d27-ebe94785eee7" (UID: "ef6bf45e-8681-41e5-8d27-ebe94785eee7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:54.180569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.180499 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6bf45e-8681-41e5-8d27-ebe94785eee7-kube-api-access-5k8df" pod "ef6bf45e-8681-41e5-8d27-ebe94785eee7" (UID: "ef6bf45e-8681-41e5-8d27-ebe94785eee7"). InnerVolumeSpecName "kube-api-access-5k8df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:54.271728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.271668 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8df\" (UniqueName: \"kubernetes.io/projected/ef6bf45e-8681-41e5-8d27-ebe94785eee7-kube-api-access-5k8df\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:54.271728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.271725 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6bf45e-8681-41e5-8d27-ebe94785eee7-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:54.371041 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.370998 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:54.571944 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.571901 4517 generic.go:356] "Generic (PLEG): container finished" podID="ef6bf45e-8681-41e5-8d27-ebe94785eee7" containerID="4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a" exitCode=0 Oct 30 01:16:54.572498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.571988 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" event={"ID":"ef6bf45e-8681-41e5-8d27-ebe94785eee7","Type":"ContainerDied","Data":"4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a"} Oct 30 01:16:54.572498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.572068 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" event={"ID":"ef6bf45e-8681-41e5-8d27-ebe94785eee7","Type":"ContainerDied","Data":"970ab26d35b68ca3f3784ccfd8f14ab86968d1a4f360a53a7d545f07735f8be1"} Oct 30 01:16:54.572498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.572001 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5" Oct 30 01:16:54.572498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.572097 4517 scope.go:122] "RemoveContainer" containerID="4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a" Oct 30 01:16:54.589003 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.588974 4517 scope.go:122] "RemoveContainer" containerID="4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a" Oct 30 01:16:54.589537 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:16:54.589500 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a\": container with ID starting with 4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a not found: ID does not exist" containerID="4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a" Oct 30 01:16:54.589626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.589540 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a"} err="failed to get container status \"4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a\": rpc error: code = NotFound desc = could not find container \"4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a\": container with ID starting with 4d5914a607c768d7ad87ccd26981b8da1bb1bfaea909c33e7c98f7229dfe294a not found: ID does not exist" Oct 30 01:16:54.625030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.624989 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5"] Oct 30 01:16:54.633919 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.633879 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-57964568c9-5tsn5"] Oct 30 01:16:54.759505 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.759471 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:54.781327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.781278 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-openshift-ca\") pod \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " Oct 30 01:16:54.781544 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.781516 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-kube-api-access-flgqs\" (UniqueName: \"kubernetes.io/projected/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-kube-api-access-flgqs\") pod \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " Oct 30 01:16:54.781673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.781632 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-xks-ca\") pod \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\" (UID: \"7c3ce1c0-ca57-48b0-a758-08177f5f0af5\") " Oct 30 01:16:54.781757 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.781680 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-openshift-ca" pod "7c3ce1c0-ca57-48b0-a758-08177f5f0af5" (UID: "7c3ce1c0-ca57-48b0-a758-08177f5f0af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:54.782488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.782462 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:54.784185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.784150 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-xks-ca" pod "7c3ce1c0-ca57-48b0-a758-08177f5f0af5" (UID: "7c3ce1c0-ca57-48b0-a758-08177f5f0af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:16:54.784649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.784623 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-kube-api-access-flgqs" pod "7c3ce1c0-ca57-48b0-a758-08177f5f0af5" (UID: "7c3ce1c0-ca57-48b0-a758-08177f5f0af5"). InnerVolumeSpecName "kube-api-access-flgqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:54.884381 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.884281 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flgqs\" (UniqueName: \"kubernetes.io/projected/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-kube-api-access-flgqs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:54.884381 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:54.884326 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c3ce1c0-ca57-48b0-a758-08177f5f0af5-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:16:55.584931 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:55.584898 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" Oct 30 01:16:55.585408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:55.584930 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk" event={"ID":"7c3ce1c0-ca57-48b0-a758-08177f5f0af5","Type":"ContainerDied","Data":"6f4d8d3f532d014abfdd5ba20fab82c9b7829803d3fadad8c6ad7d40f3bc912d"} Oct 30 01:16:55.585408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:55.584993 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f4d8d3f532d014abfdd5ba20fab82c9b7829803d3fadad8c6ad7d40f3bc912d" Oct 30 01:16:56.082217 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:56.082184 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:56.253585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:56.253541 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6bf45e-8681-41e5-8d27-ebe94785eee7" path="/var/lib/kubelet/pods/ef6bf45e-8681-41e5-8d27-ebe94785eee7/volumes" Oct 30 01:16:57.875856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:57.875816 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:59.039846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.039806 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:59.510262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.509666 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz"] Oct 30 01:16:59.510756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.510729 4517 cpu_manager.go:414] "containerMap: removing container" podUID="7c3ce1c0-ca57-48b0-a758-08177f5f0af5" containerName="revoke-keys" Oct 30 01:16:59.510884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.510757 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3ce1c0-ca57-48b0-a758-08177f5f0af5" containerName="revoke-keys" Oct 30 01:16:59.510884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.510779 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ef6bf45e-8681-41e5-8d27-ebe94785eee7" containerName="payload-pre-processing" Oct 30 01:16:59.510884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.510786 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6bf45e-8681-41e5-8d27-ebe94785eee7" containerName="payload-pre-processing" Oct 30 01:16:59.511059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.511007 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3ce1c0-ca57-48b0-a758-08177f5f0af5" containerName="revoke-keys" Oct 30 01:16:59.511059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.511028 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6bf45e-8681-41e5-8d27-ebe94785eee7" containerName="payload-pre-processing" Oct 30 01:16:59.521027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.520991 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.539802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.536046 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz"] Oct 30 01:16:59.539802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.539190 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-38b47d-openshift-default-dockercfg-b4jc2\"" Oct 30 01:16:59.587734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.587314 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-data\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.587734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.587378 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d5f4d6-77bb-4000-b374-57eac9dced96-istiod-ca-cert\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.587734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.587455 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-socket\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.587734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.587525 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-podinfo\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.587734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.587586 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-certs\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.587734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.587644 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9gh8\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-kube-api-access-r9gh8\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.587734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.587724 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-credential-socket\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.588324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.587857 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-envoy\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.588324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.587902 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-token\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.632257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.614042 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:59.746740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.730378 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-data\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.746740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.730442 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d5f4d6-77bb-4000-b374-57eac9dced96-istiod-ca-cert\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.746740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.730517 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-socket\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.746740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.730586 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-podinfo\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.746740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.730644 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-certs\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.746740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.730697 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9gh8\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-kube-api-access-r9gh8\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.746740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.730772 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-credential-socket\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.746740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.730899 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-envoy\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.746740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.730950 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-token\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.787798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.784606 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d5f4d6-77bb-4000-b374-57eac9dced96-istiod-ca-cert\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.787798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.784903 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-socket\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.872735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.848128 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-data\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.945831 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.874086 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:59.976030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.876100 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-credential-socket\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:16:59.976327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:16:59.876331 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-certs\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:00.080796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.071213 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-podinfo\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:00.080796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.071966 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9gh8\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-kube-api-access-r9gh8\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:00.097063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.087011 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-envoy\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:00.102776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.099755 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-token\") pod \"e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:00.159412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.159361 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:00.371058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.370893 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:00.527579 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.527542 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz"] Oct 30 01:17:00.527740 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:17:00.527615 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d5f4d6_77bb_4000_b374_57eac9dced96.slice/crio-cbc7275313fe1c2b37f4207ab9728e7cf7f016465a27918e00185122dc7c9e23 WatchSource:0}: Error finding container cbc7275313fe1c2b37f4207ab9728e7cf7f016465a27918e00185122dc7c9e23: Status 404 returned error can't find the container with id cbc7275313fe1c2b37f4207ab9728e7cf7f016465a27918e00185122dc7c9e23 Oct 30 01:17:00.531033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.530992 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:17:00.531158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.531069 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:17:00.531158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.531105 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:17:00.776955 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:17:00.776829 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:17:00.960144 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.960098 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" event={"ID":"66d5f4d6-77bb-4000-b374-57eac9dced96","Type":"ContainerStarted","Data":"35b4b4c03a4cfe8e9e040be1a2a9f9e549d60db0e35d2c719de82373b5e6d1b9"} Oct 30 01:17:00.960372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:00.960177 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" event={"ID":"66d5f4d6-77bb-4000-b374-57eac9dced96","Type":"ContainerStarted","Data":"cbc7275313fe1c2b37f4207ab9728e7cf7f016465a27918e00185122dc7c9e23"} Oct 30 01:17:01.027932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:01.027810 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" podStartSLOduration=2.027791391 podStartE2EDuration="2.027791391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:01.025523899 +0000 UTC m=+2558.937651040" watchObservedRunningTime="2026-10-30 01:17:01.027791391 +0000 UTC m=+2558.939918531" Oct 30 01:17:01.163821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:01.163784 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:01.191041 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:01.191007 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:01.435137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:01.435096 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:01.983882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:01.982161 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:01.984291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:01.984265 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:17:02.018466 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:02.018426 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:03.472585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:03.472539 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:03.668459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:03.668399 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:05.378806 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:05.378763 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:05.653792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:05.650606 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:05.839143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:05.839098 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c"] Oct 30 01:17:06.035912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.033968 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c"] Oct 30 01:17:06.043732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.034246 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:06.058662 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.039969 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:06.159950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.158978 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-38b47d\"" Oct 30 01:17:06.159950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.159824 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-38b47d-dockercfg-nmfq2\"" Oct 30 01:17:06.274172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.274131 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhxn\" (UniqueName: \"kubernetes.io/projected/b3ad5174-8bc1-4097-a148-5a84af54c3e6-kube-api-access-sbhxn\") pod \"payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c\" (UID: \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:06.274373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.274328 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ad5174-8bc1-4097-a148-5a84af54c3e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c\" (UID: \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:06.376231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.376126 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhxn\" (UniqueName: \"kubernetes.io/projected/b3ad5174-8bc1-4097-a148-5a84af54c3e6-kube-api-access-sbhxn\") pod \"payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c\" (UID: \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:06.376415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.376284 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ad5174-8bc1-4097-a148-5a84af54c3e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c\" (UID: \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:06.377292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.377264 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ad5174-8bc1-4097-a148-5a84af54c3e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c\" (UID: \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:06.399196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.399157 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhxn\" (UniqueName: \"kubernetes.io/projected/b3ad5174-8bc1-4097-a148-5a84af54c3e6-kube-api-access-sbhxn\") pod \"payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c\" (UID: \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:06.485592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.485541 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:06.814202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:06.814164 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c"] Oct 30 01:17:06.814531 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:17:06.814508 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ad5174_8bc1_4097_a148_5a84af54c3e6.slice/crio-21f4d8e385881bf5126ebb4fc52926efaad35e2d0ea344078d450a030eb98bd0 WatchSource:0}: Error finding container 21f4d8e385881bf5126ebb4fc52926efaad35e2d0ea344078d450a030eb98bd0: Status 404 returned error can't find the container with id 21f4d8e385881bf5126ebb4fc52926efaad35e2d0ea344078d450a030eb98bd0 Oct 30 01:17:07.184510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:07.184401 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" event={"ID":"b3ad5174-8bc1-4097-a148-5a84af54c3e6","Type":"ContainerStarted","Data":"90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f"} Oct 30 01:17:07.184510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:07.184473 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" event={"ID":"b3ad5174-8bc1-4097-a148-5a84af54c3e6","Type":"ContainerStarted","Data":"21f4d8e385881bf5126ebb4fc52926efaad35e2d0ea344078d450a030eb98bd0"} Oct 30 01:17:07.184510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:07.184500 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:07.235115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:07.235031 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" podStartSLOduration=2.2350128910000002 podStartE2EDuration="2.235012891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:07.230021109 +0000 UTC m=+2565.142148249" watchObservedRunningTime="2026-10-30 01:17:07.235012891 +0000 UTC m=+2565.147140033" Oct 30 01:17:08.008737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:08.006328 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:09.901400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:09.901358 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:11.580373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:11.580330 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:11.699360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:11.699321 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:12.797982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:12.797939 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:12.929140 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:12.929099 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:13.407381 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:13.407342 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:13.914105 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:13.914063 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:14.592253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:14.592224 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:15.021204 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:15.021080 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:15.687615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:15.687569 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:17.380161 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:17.380124 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:17.883584 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:17.883528 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:18.197837 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:18.197687 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:17:19.233642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:19.233596 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:20.891688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:20.891646 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:22.458995 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:22.458946 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:24.296002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:24.295964 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:25.549593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:25.549544 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:26.087790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.087751 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:26.531816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.531728 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72"] Oct 30 01:17:26.539543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.539502 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72"] Oct 30 01:17:26.539794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.539647 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:26.552940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.552908 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-abf8ae\"" Oct 30 01:17:26.553451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.553388 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bczgt\" (UniqueName: \"kubernetes.io/projected/038d98e9-ef10-42e7-8bda-91dfe556550f-kube-api-access-bczgt\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72\" (UID: \"038d98e9-ef10-42e7-8bda-91dfe556550f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:26.553603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.553584 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038d98e9-ef10-42e7-8bda-91dfe556550f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72\" (UID: \"038d98e9-ef10-42e7-8bda-91dfe556550f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:26.563259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.563209 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-abf8ae-dockercfg-dhssg\"" Oct 30 01:17:26.655595 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.655533 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038d98e9-ef10-42e7-8bda-91dfe556550f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72\" (UID: \"038d98e9-ef10-42e7-8bda-91dfe556550f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:26.655851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.655816 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczgt\" (UniqueName: \"kubernetes.io/projected/038d98e9-ef10-42e7-8bda-91dfe556550f-kube-api-access-bczgt\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72\" (UID: \"038d98e9-ef10-42e7-8bda-91dfe556550f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:26.656315 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.656290 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038d98e9-ef10-42e7-8bda-91dfe556550f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72\" (UID: \"038d98e9-ef10-42e7-8bda-91dfe556550f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:26.678486 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.678452 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczgt\" (UniqueName: \"kubernetes.io/projected/038d98e9-ef10-42e7-8bda-91dfe556550f-kube-api-access-bczgt\") pod \"payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72\" (UID: \"038d98e9-ef10-42e7-8bda-91dfe556550f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:26.822605 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.822533 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:26.876734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:26.874041 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:27.265896 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:27.265859 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72"] Oct 30 01:17:27.267412 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:17:27.267380 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038d98e9_ef10_42e7_8bda_91dfe556550f.slice/crio-8d217d0387281e8513ca12ffb704c4f75241c5de133961e8ec2033908f85e1b1 WatchSource:0}: Error finding container 8d217d0387281e8513ca12ffb704c4f75241c5de133961e8ec2033908f85e1b1: Status 404 returned error can't find the container with id 8d217d0387281e8513ca12ffb704c4f75241c5de133961e8ec2033908f85e1b1 Oct 30 01:17:27.270524 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:27.270499 4517 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:17:27.549510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:27.549469 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" event={"ID":"038d98e9-ef10-42e7-8bda-91dfe556550f","Type":"ContainerStarted","Data":"8d217d0387281e8513ca12ffb704c4f75241c5de133961e8ec2033908f85e1b1"} Oct 30 01:17:28.471370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:28.471334 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:29.619204 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:29.619161 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" event={"ID":"038d98e9-ef10-42e7-8bda-91dfe556550f","Type":"ContainerStarted","Data":"c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585"} Oct 30 01:17:29.619673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:29.619233 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:29.658846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:29.658772 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" podStartSLOduration=3.353700345 podStartE2EDuration="4.658748287s" totalImagesPullingTime="1.305047942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:29.656726043 +0000 UTC m=+2587.568853178" watchObservedRunningTime="2026-10-30 01:17:29.658748287 +0000 UTC m=+2587.570875441" Oct 30 01:17:30.282917 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:30.282879 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:30.768004 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:17:30.767940 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:17:31.925036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:31.924994 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:34.079055 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:34.079020 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:35.952415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:35.952374 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:37.226991 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:37.226940 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:37.377043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:37.376958 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:38.476792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:38.476742 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:40.651735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:40.650929 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:17:40.695738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:40.680434 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:41.449745 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:41.448873 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:17:42.853960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:42.853924 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:44.133782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.133744 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc"] Oct 30 01:17:44.140885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.140847 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.157692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.157662 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc"] Oct 30 01:17:44.204008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.203969 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:17:44.204431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.204411 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:17:44.232006 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.231958 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6fkq\" (UniqueName: \"kubernetes.io/projected/947aa77f-e60c-40c9-b801-89155355e852-kube-api-access-k6fkq\") pod \"maas-api-revoke-sub-41db4590-2pcrc\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.232246 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.232054 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947aa77f-e60c-40c9-b801-89155355e852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41db4590-2pcrc\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.232424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.232396 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947aa77f-e60c-40c9-b801-89155355e852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41db4590-2pcrc\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.334078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.334036 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947aa77f-e60c-40c9-b801-89155355e852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41db4590-2pcrc\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.334322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.334119 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fkq\" (UniqueName: \"kubernetes.io/projected/947aa77f-e60c-40c9-b801-89155355e852-kube-api-access-k6fkq\") pod \"maas-api-revoke-sub-41db4590-2pcrc\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.334322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.334209 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947aa77f-e60c-40c9-b801-89155355e852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41db4590-2pcrc\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.335152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.335118 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947aa77f-e60c-40c9-b801-89155355e852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41db4590-2pcrc\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.336915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.336892 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947aa77f-e60c-40c9-b801-89155355e852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41db4590-2pcrc\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.356387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.356357 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fkq\" (UniqueName: \"kubernetes.io/projected/947aa77f-e60c-40c9-b801-89155355e852-kube-api-access-k6fkq\") pod \"maas-api-revoke-sub-41db4590-2pcrc\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.512050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.511951 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:44.871629 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:44.871585 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc"] Oct 30 01:17:44.897455 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:17:44.897426 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947aa77f_e60c_40c9_b801_89155355e852.slice/crio-dfb54fe572d0f8706d9ca8422e5d2a25cab3c956f4a66d79f948a56438cbdf9f WatchSource:0}: Error finding container dfb54fe572d0f8706d9ca8422e5d2a25cab3c956f4a66d79f948a56438cbdf9f: Status 404 returned error can't find the container with id dfb54fe572d0f8706d9ca8422e5d2a25cab3c956f4a66d79f948a56438cbdf9f Oct 30 01:17:45.028364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:45.028319 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:45.239342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:45.239234 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" event={"ID":"947aa77f-e60c-40c9-b801-89155355e852","Type":"ContainerStarted","Data":"dfb54fe572d0f8706d9ca8422e5d2a25cab3c956f4a66d79f948a56438cbdf9f"} Oct 30 01:17:46.249449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:46.249411 4517 generic.go:356] "Generic (PLEG): container finished" podID="947aa77f-e60c-40c9-b801-89155355e852" containerID="57d08505b8895f9d4c1a2d73180ff44b9b58b895838b46f5ee82eebd4cad6b65" exitCode=0 Oct 30 01:17:46.249927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:46.249495 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" event={"ID":"947aa77f-e60c-40c9-b801-89155355e852","Type":"ContainerDied","Data":"57d08505b8895f9d4c1a2d73180ff44b9b58b895838b46f5ee82eebd4cad6b65"} Oct 30 01:17:46.719919 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:46.719876 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:47.453046 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.453015 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:47.525891 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.525787 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947aa77f-e60c-40c9-b801-89155355e852-kube-api-access-k6fkq\" (UniqueName: \"kubernetes.io/projected/947aa77f-e60c-40c9-b801-89155355e852-kube-api-access-k6fkq\") pod \"947aa77f-e60c-40c9-b801-89155355e852\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " Oct 30 01:17:47.525891 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.525864 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947aa77f-e60c-40c9-b801-89155355e852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947aa77f-e60c-40c9-b801-89155355e852-maas-api-openshift-ca\") pod \"947aa77f-e60c-40c9-b801-89155355e852\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " Oct 30 01:17:47.526145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.526043 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947aa77f-e60c-40c9-b801-89155355e852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947aa77f-e60c-40c9-b801-89155355e852-maas-api-xks-ca\") pod \"947aa77f-e60c-40c9-b801-89155355e852\" (UID: \"947aa77f-e60c-40c9-b801-89155355e852\") " Oct 30 01:17:47.526348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.526318 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947aa77f-e60c-40c9-b801-89155355e852-maas-api-openshift-ca" pod "947aa77f-e60c-40c9-b801-89155355e852" (UID: "947aa77f-e60c-40c9-b801-89155355e852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:47.526884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.526863 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947aa77f-e60c-40c9-b801-89155355e852-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:17:47.535275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.528910 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947aa77f-e60c-40c9-b801-89155355e852-kube-api-access-k6fkq" pod "947aa77f-e60c-40c9-b801-89155355e852" (UID: "947aa77f-e60c-40c9-b801-89155355e852"). InnerVolumeSpecName "kube-api-access-k6fkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:47.535275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.531109 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947aa77f-e60c-40c9-b801-89155355e852-maas-api-xks-ca" pod "947aa77f-e60c-40c9-b801-89155355e852" (UID: "947aa77f-e60c-40c9-b801-89155355e852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:47.628722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.628668 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947aa77f-e60c-40c9-b801-89155355e852-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:17:47.628722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:47.628717 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6fkq\" (UniqueName: \"kubernetes.io/projected/947aa77f-e60c-40c9-b801-89155355e852-kube-api-access-k6fkq\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:17:48.271096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:48.271053 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" Oct 30 01:17:48.271096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:48.271087 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc" event={"ID":"947aa77f-e60c-40c9-b801-89155355e852","Type":"ContainerDied","Data":"dfb54fe572d0f8706d9ca8422e5d2a25cab3c956f4a66d79f948a56438cbdf9f"} Oct 30 01:17:48.271321 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:48.271146 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfb54fe572d0f8706d9ca8422e5d2a25cab3c956f4a66d79f948a56438cbdf9f" Oct 30 01:17:48.430670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:48.430628 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:49.452892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:49.452845 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:17:49.453316 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:49.452931 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:17:50.007725 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:50.007679 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:51.674981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:51.674904 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:52.314628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:52.314593 4517 generic.go:356] "Generic (PLEG): container finished" podID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerID="f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0" exitCode=0 Oct 30 01:17:52.314865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:52.314664 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" event={"ID":"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2","Type":"ContainerDied","Data":"f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0"} Oct 30 01:17:52.315228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:52.315213 4517 scope.go:122] "RemoveContainer" containerID="f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0" Oct 30 01:17:52.317634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:52.317602 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:17:52.317722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:52.317684 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:17:52.317766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:52.317743 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:17:53.324488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:53.324448 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" event={"ID":"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2","Type":"ContainerStarted","Data":"c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f"} Oct 30 01:17:53.520587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:53.520554 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:53.872733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:53.872166 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:17:53.872733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:53.872322 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:17:53.872733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:53.872368 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 01:17:54.446595 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.446558 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk"] Oct 30 01:17:54.447337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.447318 4517 cpu_manager.go:414] "containerMap: removing container" podUID="947aa77f-e60c-40c9-b801-89155355e852" containerName="revoke-keys" Oct 30 01:17:54.447416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.447338 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947aa77f-e60c-40c9-b801-89155355e852" containerName="revoke-keys" Oct 30 01:17:54.447487 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.447475 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="947aa77f-e60c-40c9-b801-89155355e852" containerName="revoke-keys" Oct 30 01:17:54.451387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.451369 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.467032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.467004 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:17:54.489569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.488627 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:17:54.503002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.491196 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk"] Oct 30 01:17:54.608590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.608550 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bcdde58-276qk\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.608841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.608805 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5b95\" (UniqueName: \"kubernetes.io/projected/6e267981-ea19-4f5f-be8c-9649fb997a6b-kube-api-access-b5b95\") pod \"maas-api-revoke-sub-7bcdde58-276qk\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.608991 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.608857 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bcdde58-276qk\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.709957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.709860 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bcdde58-276qk\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.710147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.710058 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bcdde58-276qk\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.710293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.710274 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5b95\" (UniqueName: \"kubernetes.io/projected/6e267981-ea19-4f5f-be8c-9649fb997a6b-kube-api-access-b5b95\") pod \"maas-api-revoke-sub-7bcdde58-276qk\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.711091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.711056 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bcdde58-276qk\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.712521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.712502 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bcdde58-276qk\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.732422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.732385 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5b95\" (UniqueName: \"kubernetes.io/projected/6e267981-ea19-4f5f-be8c-9649fb997a6b-kube-api-access-b5b95\") pod \"maas-api-revoke-sub-7bcdde58-276qk\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.791121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.791054 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:54.876924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.876297 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:17:54.876924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:54.876359 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 01:17:55.108870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:55.108837 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk"] Oct 30 01:17:55.198398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:55.198358 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:55.356047 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:55.356010 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:55.396208 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:55.396088 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" event={"ID":"6e267981-ea19-4f5f-be8c-9649fb997a6b","Type":"ContainerStarted","Data":"064fc5f7b485284700d234e8f0a64d1a94f117568c2ce5f9a78292900622afeb"} Oct 30 01:17:55.493762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:55.493114 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:55.891892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:55.891857 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:17:55.892173 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:55.892143 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 01:17:56.447262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:56.446827 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" event={"ID":"6e267981-ea19-4f5f-be8c-9649fb997a6b","Type":"ContainerStarted","Data":"f273cd7a96c7bae797b16d24e081868700fff91115af9e1585341b63ab7b040e"} Oct 30 01:17:56.535832 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:56.534284 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" podStartSLOduration=2.534261598 podStartE2EDuration="2.534261598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:56.523180854 +0000 UTC m=+2614.435308006" watchObservedRunningTime="2026-10-30 01:17:56.534261598 +0000 UTC m=+2614.446388756" Oct 30 01:17:56.871191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:56.871147 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:17:56.871498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:56.871466 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 01:17:56.923934 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:56.923902 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:57.455830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:57.455700 4517 generic.go:356] "Generic (PLEG): container finished" podID="6e267981-ea19-4f5f-be8c-9649fb997a6b" containerID="f273cd7a96c7bae797b16d24e081868700fff91115af9e1585341b63ab7b040e" exitCode=0 Oct 30 01:17:57.456012 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:57.456000 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" event={"ID":"6e267981-ea19-4f5f-be8c-9649fb997a6b","Type":"ContainerDied","Data":"f273cd7a96c7bae797b16d24e081868700fff91115af9e1585341b63ab7b040e"} Oct 30 01:17:57.881231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:57.881191 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:17:57.881971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:57.881924 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 01:17:58.720296 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.720233 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:58.750226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.750157 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:58.831862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.828292 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e267981-ea19-4f5f-be8c-9649fb997a6b-kube-api-access-b5b95\" (UniqueName: \"kubernetes.io/projected/6e267981-ea19-4f5f-be8c-9649fb997a6b-kube-api-access-b5b95\") pod \"6e267981-ea19-4f5f-be8c-9649fb997a6b\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " Oct 30 01:17:58.831862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.828423 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-xks-ca\") pod \"6e267981-ea19-4f5f-be8c-9649fb997a6b\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " Oct 30 01:17:58.831862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.828517 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-openshift-ca\") pod \"6e267981-ea19-4f5f-be8c-9649fb997a6b\" (UID: \"6e267981-ea19-4f5f-be8c-9649fb997a6b\") " Oct 30 01:17:58.831862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.830039 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-openshift-ca" pod "6e267981-ea19-4f5f-be8c-9649fb997a6b" (UID: "6e267981-ea19-4f5f-be8c-9649fb997a6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:58.926743 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.923001 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:17:58.926743 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.923141 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 01:17:58.940909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.936567 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:17:58.945783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.944390 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e267981-ea19-4f5f-be8c-9649fb997a6b-kube-api-access-b5b95" pod "6e267981-ea19-4f5f-be8c-9649fb997a6b" (UID: "6e267981-ea19-4f5f-be8c-9649fb997a6b"). InnerVolumeSpecName "kube-api-access-b5b95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:58.947350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:58.947205 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-xks-ca" pod "6e267981-ea19-4f5f-be8c-9649fb997a6b" (UID: "6e267981-ea19-4f5f-be8c-9649fb997a6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:59.038614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:59.038575 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e267981-ea19-4f5f-be8c-9649fb997a6b-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:17:59.038614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:59.038616 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5b95\" (UniqueName: \"kubernetes.io/projected/6e267981-ea19-4f5f-be8c-9649fb997a6b-kube-api-access-b5b95\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:17:59.507218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:59.507127 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" Oct 30 01:17:59.507218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:59.507139 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk" event={"ID":"6e267981-ea19-4f5f-be8c-9649fb997a6b","Type":"ContainerDied","Data":"064fc5f7b485284700d234e8f0a64d1a94f117568c2ce5f9a78292900622afeb"} Oct 30 01:17:59.507218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:59.507198 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="064fc5f7b485284700d234e8f0a64d1a94f117568c2ce5f9a78292900622afeb" Oct 30 01:17:59.877667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:59.877630 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:17:59.878106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:59.878084 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:17:59.878859 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:17:59.878838 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:18:00.355267 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:00.355233 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:00.762179 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:18:00.762073 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:18:01.947025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:01.946991 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:03.634104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:03.634028 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:05.291691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:05.291623 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:06.031514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:06.031478 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464"] Oct 30 01:18:06.042344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:06.042308 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede1b0c6-hk464"] Oct 30 01:18:06.238369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:06.238329 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="234df198-0f8d-4ed8-b6ea-6821e65631a8" path="/var/lib/kubelet/pods/234df198-0f8d-4ed8-b6ea-6821e65631a8/volumes" Oct 30 01:18:06.620116 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:06.620013 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:06.999659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:06.999559 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:07.454425 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.454385 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:07.852987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.852949 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b"] Oct 30 01:18:07.854102 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.854077 4517 cpu_manager.go:414] "containerMap: removing container" podUID="6e267981-ea19-4f5f-be8c-9649fb997a6b" containerName="revoke-keys" Oct 30 01:18:07.854167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.854106 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e267981-ea19-4f5f-be8c-9649fb997a6b" containerName="revoke-keys" Oct 30 01:18:07.854304 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.854292 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e267981-ea19-4f5f-be8c-9649fb997a6b" containerName="revoke-keys" Oct 30 01:18:07.862160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.862130 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:07.878465 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.878417 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b"] Oct 30 01:18:07.879624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.879601 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:18:07.879868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.879650 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:18:07.959431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.959384 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqd8j\" (UniqueName: \"kubernetes.io/projected/bf6d086e-9991-40d3-b387-358440a386cf-kube-api-access-cqd8j\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:07.959652 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.959554 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:07.959795 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:07.959762 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:08.062240 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.062195 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqd8j\" (UniqueName: \"kubernetes.io/projected/bf6d086e-9991-40d3-b387-358440a386cf-kube-api-access-cqd8j\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:08.062461 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.062309 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:08.062461 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.062401 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:08.063128 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.063101 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:08.065292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.065261 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:08.084605 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.084569 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqd8j\" (UniqueName: \"kubernetes.io/projected/bf6d086e-9991-40d3-b387-358440a386cf-kube-api-access-cqd8j\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:08.187893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.187794 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:08.510005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.509966 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b"] Oct 30 01:18:08.511287 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:18:08.511260 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6d086e_9991_40d3_b387_358440a386cf.slice/crio-e5ea525435d78056bf2819c7b8e6c760ea42faef327eac7b680902164f4bcad7 WatchSource:0}: Error finding container e5ea525435d78056bf2819c7b8e6c760ea42faef327eac7b680902164f4bcad7: Status 404 returned error can't find the container with id e5ea525435d78056bf2819c7b8e6c760ea42faef327eac7b680902164f4bcad7 Oct 30 01:18:08.637502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.637450 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" event={"ID":"bf6d086e-9991-40d3-b387-358440a386cf","Type":"ContainerStarted","Data":"e5ea525435d78056bf2819c7b8e6c760ea42faef327eac7b680902164f4bcad7"} Oct 30 01:18:08.912297 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:08.912246 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:09.650178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:09.650129 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" event={"ID":"bf6d086e-9991-40d3-b387-358440a386cf","Type":"ContainerStarted","Data":"ad70b0e37a8acf68d15af933f773e8a50e1f7498ae1f70284d58738d46ef3746"} Oct 30 01:18:09.692236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:09.692172 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" podStartSLOduration=2.692156441 podStartE2EDuration="2.692156441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:09.687345974 +0000 UTC m=+2627.599473116" watchObservedRunningTime="2026-10-30 01:18:09.692156441 +0000 UTC m=+2627.604283628" Oct 30 01:18:10.030906 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:10.030797 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt"] Oct 30 01:18:10.036022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:10.035980 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59216e08-96gbt"] Oct 30 01:18:10.238846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:10.238814 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ae8413-1d69-4bbd-a2b8-ad7875dd147b" path="/var/lib/kubelet/pods/c7ae8413-1d69-4bbd-a2b8-ad7875dd147b/volumes" Oct 30 01:18:10.598319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:10.598267 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:10.688951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:10.688918 4517 generic.go:356] "Generic (PLEG): container finished" podID="bf6d086e-9991-40d3-b387-358440a386cf" containerID="ad70b0e37a8acf68d15af933f773e8a50e1f7498ae1f70284d58738d46ef3746" exitCode=0 Oct 30 01:18:10.689158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:10.689135 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" event={"ID":"bf6d086e-9991-40d3-b387-358440a386cf","Type":"ContainerDied","Data":"ad70b0e37a8acf68d15af933f773e8a50e1f7498ae1f70284d58738d46ef3746"} Oct 30 01:18:11.925887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:11.925852 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:12.041443 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.041412 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-openshift-ca\") pod \"bf6d086e-9991-40d3-b387-358440a386cf\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " Oct 30 01:18:12.041673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.041528 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-xks-ca\") pod \"bf6d086e-9991-40d3-b387-358440a386cf\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " Oct 30 01:18:12.041783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.041681 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6d086e-9991-40d3-b387-358440a386cf-kube-api-access-cqd8j\" (UniqueName: \"kubernetes.io/projected/bf6d086e-9991-40d3-b387-358440a386cf-kube-api-access-cqd8j\") pod \"bf6d086e-9991-40d3-b387-358440a386cf\" (UID: \"bf6d086e-9991-40d3-b387-358440a386cf\") " Oct 30 01:18:12.041895 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.041864 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-openshift-ca" pod "bf6d086e-9991-40d3-b387-358440a386cf" (UID: "bf6d086e-9991-40d3-b387-358440a386cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:12.042497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.042479 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:18:12.044229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.044193 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6d086e-9991-40d3-b387-358440a386cf-kube-api-access-cqd8j" pod "bf6d086e-9991-40d3-b387-358440a386cf" (UID: "bf6d086e-9991-40d3-b387-358440a386cf"). InnerVolumeSpecName "kube-api-access-cqd8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:12.044383 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.044365 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-xks-ca" pod "bf6d086e-9991-40d3-b387-358440a386cf" (UID: "bf6d086e-9991-40d3-b387-358440a386cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:12.144531 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.144488 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d086e-9991-40d3-b387-358440a386cf-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:18:12.144531 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.144527 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqd8j\" (UniqueName: \"kubernetes.io/projected/bf6d086e-9991-40d3-b387-358440a386cf-kube-api-access-cqd8j\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:18:12.347224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.347109 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:12.708278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.708193 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" event={"ID":"bf6d086e-9991-40d3-b387-358440a386cf","Type":"ContainerDied","Data":"e5ea525435d78056bf2819c7b8e6c760ea42faef327eac7b680902164f4bcad7"} Oct 30 01:18:12.708278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.708230 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b" Oct 30 01:18:12.708278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:12.708250 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ea525435d78056bf2819c7b8e6c760ea42faef327eac7b680902164f4bcad7" Oct 30 01:18:14.063554 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:14.063518 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:15.685173 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:15.685130 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:16.030476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:16.030387 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn"] Oct 30 01:18:16.041316 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:16.041283 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b638c49-dnnhn"] Oct 30 01:18:16.237496 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:16.237457 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c" path="/var/lib/kubelet/pods/4d16d2e9-91f8-4eca-bd98-2ad3dd604c3c/volumes" Oct 30 01:18:17.843426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:17.843390 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:18.086421 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:18.086383 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:18.425622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:18.425582 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:18.578773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:18.578737 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:19.592202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:19.592134 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:20.035183 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:20.035152 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj"] Oct 30 01:18:20.045525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:20.045483 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b4af30-bqzmj"] Oct 30 01:18:20.237406 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:20.237372 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dee994c-fb6e-4d1b-bd07-314f8df320d2" path="/var/lib/kubelet/pods/4dee994c-fb6e-4d1b-bd07-314f8df320d2/volumes" Oct 30 01:18:21.192807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:21.192760 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:22.837473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:22.837434 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:25.101828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:25.101787 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:26.743599 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:26.743562 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:29.160325 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:29.160282 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:31.039534 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:18:31.039473 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:18:31.509785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:31.509609 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:33.152647 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:33.152550 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:35.282785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:35.282692 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:36.910308 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:36.910251 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:38.526671 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:38.526634 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:40.164964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:40.164920 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:42.016306 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:42.016272 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:42.149701 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:42.149653 4517 generic.go:356] "Generic (PLEG): container finished" podID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerID="35b4b4c03a4cfe8e9e040be1a2a9f9e549d60db0e35d2c719de82373b5e6d1b9" exitCode=0 Oct 30 01:18:42.149701 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:42.149759 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" event={"ID":"66d5f4d6-77bb-4000-b374-57eac9dced96","Type":"ContainerDied","Data":"35b4b4c03a4cfe8e9e040be1a2a9f9e549d60db0e35d2c719de82373b5e6d1b9"} Oct 30 01:18:42.150281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:42.150266 4517 scope.go:122] "RemoveContainer" containerID="35b4b4c03a4cfe8e9e040be1a2a9f9e549d60db0e35d2c719de82373b5e6d1b9" Oct 30 01:18:42.152339 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:42.152310 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:18:42.152422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:42.152380 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:18:42.152422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:42.152416 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:18:43.030085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:43.030051 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn"] Oct 30 01:18:43.040905 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:43.040878 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c522237d-hmqcn"] Oct 30 01:18:43.160435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:43.160394 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" event={"ID":"66d5f4d6-77bb-4000-b374-57eac9dced96","Type":"ContainerStarted","Data":"8b9c7140e4b3374c50c4a494c4266a6798859968c3374461d171b3d820183eec"} Oct 30 01:18:43.659048 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:43.658996 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:44.163010 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:44.162043 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:18:44.164551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:44.164521 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:18:44.164782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:44.164753 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:18:44.236873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:44.236841 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df32418-cac6-433c-a7d6-285e12a053f4" path="/var/lib/kubelet/pods/6df32418-cac6-433c-a7d6-285e12a053f4/volumes" Oct 30 01:18:44.252542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:44.252499 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:45.162104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:45.162066 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:18:45.162429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:45.162393 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:18:45.442753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:45.442627 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:46.177732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:46.173209 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:18:46.177732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:46.173287 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:18:46.991408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:46.990844 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:18:46.997096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:46.997069 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:47.171885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:47.171844 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:18:47.172282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:47.172244 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:18:48.004934 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:48.004892 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:48.162742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:48.161062 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:18:48.162742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:48.161134 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:18:48.811951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:48.811913 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:49.166103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:49.166018 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:18:49.167406 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:49.167383 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:18:50.470299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:50.470259 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:52.110830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:52.110786 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:53.034159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:53.034118 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5"] Oct 30 01:18:53.045563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:53.045520 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451bb244-z4vc5"] Oct 30 01:18:53.834827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:53.834789 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:54.244171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:54.244089 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67717cb8-67ca-422d-ac77-77be7d572941" path="/var/lib/kubelet/pods/67717cb8-67ca-422d-ac77-77be7d572941/volumes" Oct 30 01:18:55.384107 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:55.384065 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:56.976076 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:56.976042 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:58.767652 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:18:58.767609 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:00.354865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:00.354832 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:00.769242 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:00.769110 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:19:01.919089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:01.919055 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:03.031841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:03.031798 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws"] Oct 30 01:19:03.042369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:03.042331 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46308dd8-n65ws"] Oct 30 01:19:03.356430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:03.356384 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:04.218816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:04.218772 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:04.271344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:04.271306 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a" path="/var/lib/kubelet/pods/4eaf3f59-b4c7-4a15-bf00-216d8a3e0a9a/volumes" Oct 30 01:19:04.344616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:04.344578 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:04.378541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:04.378504 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b"] Oct 30 01:19:04.404383 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:04.404227 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-320045-mnv6b"] Oct 30 01:19:04.469048 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:04.468942 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:04.846214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:04.846180 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:06.238463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:06.238424 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6d086e-9991-40d3-b387-358440a386cf" path="/var/lib/kubelet/pods/bf6d086e-9991-40d3-b387-358440a386cf/volumes" Oct 30 01:19:06.815206 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:06.815161 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:08.629872 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:08.629837 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:10.183539 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:10.183501 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:11.017489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.017448 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:11.139383 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.139333 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:11.356681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.356353 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb"] Oct 30 01:19:11.364202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.357491 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" containerID="cri-o://3a4f6c81a36a27695bf90cc2101a037c0109bba926a1bdddea78d49ccb98a143" gracePeriod=30 Oct 30 01:19:11.371597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.367012 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:11.383798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.383333 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b"] Oct 30 01:19:11.383798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.383583 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" podUID="3f823884-3933-465d-a308-c27be99ef1a3" containerName="istio-proxy" containerID="cri-o://d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a" gracePeriod=30 Oct 30 01:19:11.945441 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.945402 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7"] Oct 30 01:19:11.945794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.945761 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" containerID="cri-o://29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f" gracePeriod=30 Oct 30 01:19:11.970079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.970036 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr"] Oct 30 01:19:11.970773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.970697 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" containerID="cri-o://a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438" gracePeriod=30 Oct 30 01:19:11.995901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.995863 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb"] Oct 30 01:19:11.996624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.996608 4517 cpu_manager.go:414] "containerMap: removing container" podUID="bf6d086e-9991-40d3-b387-358440a386cf" containerName="revoke-keys" Oct 30 01:19:11.996679 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.996626 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6d086e-9991-40d3-b387-358440a386cf" containerName="revoke-keys" Oct 30 01:19:11.996794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:11.996770 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6d086e-9991-40d3-b387-358440a386cf" containerName="revoke-keys" Oct 30 01:19:12.002039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.002010 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.003192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.003153 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb"] Oct 30 01:19:12.017416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.017382 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:19:12.017660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.017424 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:19:12.100041 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.100002 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9v4\" (UniqueName: \"kubernetes.io/projected/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-kube-api-access-hf9v4\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.100271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.100245 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.100419 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.100398 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.188146 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.188107 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:12.202374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.202286 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.202530 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.202444 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.202644 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.202627 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9v4\" (UniqueName: \"kubernetes.io/projected/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-kube-api-access-hf9v4\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.203099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.203067 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.205333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.205304 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.275730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.268661 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9v4\" (UniqueName: \"kubernetes.io/projected/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-kube-api-access-hf9v4\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.319774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.319731 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:12.638000 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:12.637968 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722ffa59_fa18_488c_9a39_5a4f0d38b5c2.slice/crio-a7c8a7e6a86413275494a5398001131a92f8841970898252d421af1c367d8b8c WatchSource:0}: Error finding container a7c8a7e6a86413275494a5398001131a92f8841970898252d421af1c367d8b8c: Status 404 returned error can't find the container with id a7c8a7e6a86413275494a5398001131a92f8841970898252d421af1c367d8b8c Oct 30 01:19:12.638596 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.638570 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb"] Oct 30 01:19:12.797561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:12.797521 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" event={"ID":"722ffa59-fa18-488c-9a39-5a4f0d38b5c2","Type":"ContainerStarted","Data":"a7c8a7e6a86413275494a5398001131a92f8841970898252d421af1c367d8b8c"} Oct 30 01:19:13.810819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:13.810775 4517 generic.go:356] "Generic (PLEG): container finished" podID="722ffa59-fa18-488c-9a39-5a4f0d38b5c2" containerID="6ceade41acbe4d21d519b6a5244b6dec75da925b79ad318047e45c7dd39ac55c" exitCode=0 Oct 30 01:19:13.811274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:13.810864 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" event={"ID":"722ffa59-fa18-488c-9a39-5a4f0d38b5c2","Type":"ContainerDied","Data":"6ceade41acbe4d21d519b6a5244b6dec75da925b79ad318047e45c7dd39ac55c"} Oct 30 01:19:14.030178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:14.030086 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx"] Oct 30 01:19:14.034642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:14.034609 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e7a0a6-7hqgx"] Oct 30 01:19:14.238753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:14.238671 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c51e7f8-ead6-4b99-be17-d179093cd9f6" path="/var/lib/kubelet/pods/9c51e7f8-ead6-4b99-be17-d179093cd9f6/volumes" Oct 30 01:19:14.658204 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:14.658168 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:15.014888 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.014858 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:15.162281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.162231 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-openshift-ca\") pod \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " Oct 30 01:19:15.162525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.162310 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-xks-ca\") pod \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " Oct 30 01:19:15.162625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.162610 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-kube-api-access-hf9v4\" (UniqueName: \"kubernetes.io/projected/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-kube-api-access-hf9v4\") pod \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\" (UID: \"722ffa59-fa18-488c-9a39-5a4f0d38b5c2\") " Oct 30 01:19:15.162807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.162696 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-openshift-ca" pod "722ffa59-fa18-488c-9a39-5a4f0d38b5c2" (UID: "722ffa59-fa18-488c-9a39-5a4f0d38b5c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:15.163453 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.163419 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:15.165246 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.165212 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-kube-api-access-hf9v4" pod "722ffa59-fa18-488c-9a39-5a4f0d38b5c2" (UID: "722ffa59-fa18-488c-9a39-5a4f0d38b5c2"). InnerVolumeSpecName "kube-api-access-hf9v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:15.165363 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.165322 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-xks-ca" pod "722ffa59-fa18-488c-9a39-5a4f0d38b5c2" (UID: "722ffa59-fa18-488c-9a39-5a4f0d38b5c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:15.266039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.265924 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9v4\" (UniqueName: \"kubernetes.io/projected/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-kube-api-access-hf9v4\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:15.266039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.265983 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/722ffa59-fa18-488c-9a39-5a4f0d38b5c2-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:15.844293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.844256 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" Oct 30 01:19:15.844293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.844279 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb" event={"ID":"722ffa59-fa18-488c-9a39-5a4f0d38b5c2","Type":"ContainerDied","Data":"a7c8a7e6a86413275494a5398001131a92f8841970898252d421af1c367d8b8c"} Oct 30 01:19:15.844540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:15.844357 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7c8a7e6a86413275494a5398001131a92f8841970898252d421af1c367d8b8c" Oct 30 01:19:16.673951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.673824 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:16.858572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.858540 4517 generic.go:356] "Generic (PLEG): container finished" podID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerID="3a4f6c81a36a27695bf90cc2101a037c0109bba926a1bdddea78d49ccb98a143" exitCode=0 Oct 30 01:19:16.858878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.858619 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" event={"ID":"336252d7-b778-490b-93e8-7f2c4ca5db60","Type":"ContainerDied","Data":"3a4f6c81a36a27695bf90cc2101a037c0109bba926a1bdddea78d49ccb98a143"} Oct 30 01:19:16.859073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.859048 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" event={"ID":"336252d7-b778-490b-93e8-7f2c4ca5db60","Type":"ContainerDied","Data":"0ac3fe19d3764f9b401847863f130c88f1cd34a4a288acc8beb8846ef8bf7689"} Oct 30 01:19:16.859139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.859089 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ac3fe19d3764f9b401847863f130c88f1cd34a4a288acc8beb8846ef8bf7689" Oct 30 01:19:16.886553 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.885973 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:19:16.927595 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.927101 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-kube-api-access-784sg\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-kube-api-access-784sg\") pod \"336252d7-b778-490b-93e8-7f2c4ca5db60\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " Oct 30 01:19:16.928394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.927986 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336252d7-b778-490b-93e8-7f2c4ca5db60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336252d7-b778-490b-93e8-7f2c4ca5db60-istiod-ca-cert\") pod \"336252d7-b778-490b-93e8-7f2c4ca5db60\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " Oct 30 01:19:16.928394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.928073 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-credential-socket\") pod \"336252d7-b778-490b-93e8-7f2c4ca5db60\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " Oct 30 01:19:16.928394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.928234 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-socket\") pod \"336252d7-b778-490b-93e8-7f2c4ca5db60\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " Oct 30 01:19:16.928394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.928266 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-token\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-token\") pod \"336252d7-b778-490b-93e8-7f2c4ca5db60\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " Oct 30 01:19:16.928394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.928328 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-certs\") pod \"336252d7-b778-490b-93e8-7f2c4ca5db60\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " Oct 30 01:19:16.929806 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.928366 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-podinfo\") pod \"336252d7-b778-490b-93e8-7f2c4ca5db60\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " Oct 30 01:19:16.929806 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.928876 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-envoy\") pod \"336252d7-b778-490b-93e8-7f2c4ca5db60\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " Oct 30 01:19:16.929806 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.928941 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-data\") pod \"336252d7-b778-490b-93e8-7f2c4ca5db60\" (UID: \"336252d7-b778-490b-93e8-7f2c4ca5db60\") " Oct 30 01:19:16.931008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.930648 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-data" pod "336252d7-b778-490b-93e8-7f2c4ca5db60" (UID: "336252d7-b778-490b-93e8-7f2c4ca5db60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.931537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.931310 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336252d7-b778-490b-93e8-7f2c4ca5db60-istiod-ca-cert" pod "336252d7-b778-490b-93e8-7f2c4ca5db60" (UID: "336252d7-b778-490b-93e8-7f2c4ca5db60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:16.931862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.931679 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-credential-socket" pod "336252d7-b778-490b-93e8-7f2c4ca5db60" (UID: "336252d7-b778-490b-93e8-7f2c4ca5db60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.932121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.931990 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-socket" pod "336252d7-b778-490b-93e8-7f2c4ca5db60" (UID: "336252d7-b778-490b-93e8-7f2c4ca5db60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.932121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.932013 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-certs" pod "336252d7-b778-490b-93e8-7f2c4ca5db60" (UID: "336252d7-b778-490b-93e8-7f2c4ca5db60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.948649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.946085 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-kube-api-access-784sg" pod "336252d7-b778-490b-93e8-7f2c4ca5db60" (UID: "336252d7-b778-490b-93e8-7f2c4ca5db60"). InnerVolumeSpecName "kube-api-access-784sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:16.957036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.956971 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-podinfo" pod "336252d7-b778-490b-93e8-7f2c4ca5db60" (UID: "336252d7-b778-490b-93e8-7f2c4ca5db60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:16.957571 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.957529 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-envoy" pod "336252d7-b778-490b-93e8-7f2c4ca5db60" (UID: "336252d7-b778-490b-93e8-7f2c4ca5db60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.985986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.982576 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m"] Oct 30 01:19:16.985986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.983980 4517 cpu_manager.go:414] "containerMap: removing container" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" Oct 30 01:19:16.985986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.984006 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" Oct 30 01:19:16.985986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.984026 4517 cpu_manager.go:414] "containerMap: removing container" podUID="722ffa59-fa18-488c-9a39-5a4f0d38b5c2" containerName="revoke-keys" Oct 30 01:19:16.985986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.984034 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722ffa59-fa18-488c-9a39-5a4f0d38b5c2" containerName="revoke-keys" Oct 30 01:19:16.985986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.984135 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-token" pod "336252d7-b778-490b-93e8-7f2c4ca5db60" (UID: "336252d7-b778-490b-93e8-7f2c4ca5db60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:16.985986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.984287 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" containerName="istio-proxy" Oct 30 01:19:16.985986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.984301 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="722ffa59-fa18-488c-9a39-5a4f0d38b5c2" containerName="revoke-keys" Oct 30 01:19:16.990685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:16.990498 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.008305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.007902 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m"] Oct 30 01:19:17.031401 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031366 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9xp7\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-kube-api-access-z9xp7\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.031590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031457 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.031590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031499 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-certs\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.031590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031517 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-credential-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.031590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031536 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cf5570-1952-4038-af67-dded46578a90-istio-podinfo\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.031590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031570 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-envoy\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.031590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031589 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-istio-token\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031621 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cf5570-1952-4038-af67-dded46578a90-istiod-ca-cert\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031732 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-data\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031828 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031839 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031849 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031861 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031874 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031886 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031894 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336252d7-b778-490b-93e8-7f2c4ca5db60-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031903 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-784sg\" (UniqueName: \"kubernetes.io/projected/336252d7-b778-490b-93e8-7f2c4ca5db60-kube-api-access-784sg\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.031950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.031911 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336252d7-b778-490b-93e8-7f2c4ca5db60-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.103927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.103471 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5f99e7-openshift-default-dockercfg-lqqpl\"" Oct 30 01:19:17.133355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.133300 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9xp7\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-kube-api-access-z9xp7\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.133588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.133461 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.133588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.133534 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-certs\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.133588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.133557 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-credential-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.133588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.133579 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cf5570-1952-4038-af67-dded46578a90-istio-podinfo\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.133837 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.133645 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-envoy\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.133837 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.133677 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-istio-token\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.134340 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.134233 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-certs\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.136025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.134631 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cf5570-1952-4038-af67-dded46578a90-istiod-ca-cert\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.136025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.134880 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-data\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.136025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.135499 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-data\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.137098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.136743 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-credential-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.137627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.137577 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.137921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.137618 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cf5570-1952-4038-af67-dded46578a90-istiod-ca-cert\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.139804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.139743 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-envoy\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.141482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.141420 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cf5570-1952-4038-af67-dded46578a90-istio-podinfo\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.164946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.163924 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-istio-token\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.177902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.177820 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9xp7\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-kube-api-access-z9xp7\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.198022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.197986 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:19:17.236178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236139 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-data\") pod \"3f823884-3933-465d-a308-c27be99ef1a3\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " Oct 30 01:19:17.236383 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236220 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-istio-token\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-istio-token\") pod \"3f823884-3933-465d-a308-c27be99ef1a3\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " Oct 30 01:19:17.236383 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236339 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-certs\") pod \"3f823884-3933-465d-a308-c27be99ef1a3\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " Oct 30 01:19:17.236516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236404 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-credential-socket\") pod \"3f823884-3933-465d-a308-c27be99ef1a3\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " Oct 30 01:19:17.236516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236451 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-data" pod "3f823884-3933-465d-a308-c27be99ef1a3" (UID: "3f823884-3933-465d-a308-c27be99ef1a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.236516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236456 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f823884-3933-465d-a308-c27be99ef1a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f823884-3933-465d-a308-c27be99ef1a3-istio-podinfo\") pod \"3f823884-3933-465d-a308-c27be99ef1a3\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " Oct 30 01:19:17.236700 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236548 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-kube-api-access-xxvw6\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-kube-api-access-xxvw6\") pod \"3f823884-3933-465d-a308-c27be99ef1a3\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " Oct 30 01:19:17.236700 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236642 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-envoy\") pod \"3f823884-3933-465d-a308-c27be99ef1a3\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " Oct 30 01:19:17.236836 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236695 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-socket\") pod \"3f823884-3933-465d-a308-c27be99ef1a3\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " Oct 30 01:19:17.236836 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236807 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f823884-3933-465d-a308-c27be99ef1a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f823884-3933-465d-a308-c27be99ef1a3-istiod-ca-cert\") pod \"3f823884-3933-465d-a308-c27be99ef1a3\" (UID: \"3f823884-3933-465d-a308-c27be99ef1a3\") " Oct 30 01:19:17.236954 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.236843 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-certs" pod "3f823884-3933-465d-a308-c27be99ef1a3" (UID: "3f823884-3933-465d-a308-c27be99ef1a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.237177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.237042 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-socket" pod "3f823884-3933-465d-a308-c27be99ef1a3" (UID: "3f823884-3933-465d-a308-c27be99ef1a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.237177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.237142 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-credential-socket" pod "3f823884-3933-465d-a308-c27be99ef1a3" (UID: "3f823884-3933-465d-a308-c27be99ef1a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.237893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.237308 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f823884-3933-465d-a308-c27be99ef1a3-istiod-ca-cert" pod "3f823884-3933-465d-a308-c27be99ef1a3" (UID: "3f823884-3933-465d-a308-c27be99ef1a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:17.237893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.237744 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.237893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.237766 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f823884-3933-465d-a308-c27be99ef1a3-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.237893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.237779 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.237893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.237792 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.237893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.237803 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.240504 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.240465 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-istio-token" pod "3f823884-3933-465d-a308-c27be99ef1a3" (UID: "3f823884-3933-465d-a308-c27be99ef1a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:17.241251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.241205 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-kube-api-access-xxvw6" pod "3f823884-3933-465d-a308-c27be99ef1a3" (UID: "3f823884-3933-465d-a308-c27be99ef1a3"). InnerVolumeSpecName "kube-api-access-xxvw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:17.241524 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.241493 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f823884-3933-465d-a308-c27be99ef1a3-istio-podinfo" pod "3f823884-3933-465d-a308-c27be99ef1a3" (UID: "3f823884-3933-465d-a308-c27be99ef1a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:17.243088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.243057 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-envoy" pod "3f823884-3933-465d-a308-c27be99ef1a3" (UID: "3f823884-3933-465d-a308-c27be99ef1a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.334191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.334157 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:19:17.339469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.339434 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f823884-3933-465d-a308-c27be99ef1a3-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.339469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.339470 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxvw6\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-kube-api-access-xxvw6\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.339742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.339486 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f823884-3933-465d-a308-c27be99ef1a3-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.339742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.339503 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f823884-3933-465d-a308-c27be99ef1a3-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.405329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.405295 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:19:17.438120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.438083 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:17.440415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440369 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-socket\") pod \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " Oct 30 01:19:17.440619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440463 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-data\") pod \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " Oct 30 01:19:17.440619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440528 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-certs\") pod \"b39d793a-45dc-4682-b068-7666d06bb566\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " Oct 30 01:19:17.440619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440578 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-envoy\") pod \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " Oct 30 01:19:17.440845 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440635 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istiod-ca-cert\") pod \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " Oct 30 01:19:17.440845 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440676 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-istio-token\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-istio-token\") pod \"b39d793a-45dc-4682-b068-7666d06bb566\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " Oct 30 01:19:17.440845 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440754 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-envoy\") pod \"b39d793a-45dc-4682-b068-7666d06bb566\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " Oct 30 01:19:17.440845 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440783 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-socket\") pod \"b39d793a-45dc-4682-b068-7666d06bb566\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " Oct 30 01:19:17.441073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440896 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39d793a-45dc-4682-b068-7666d06bb566-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d793a-45dc-4682-b068-7666d06bb566-istiod-ca-cert\") pod \"b39d793a-45dc-4682-b068-7666d06bb566\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " Oct 30 01:19:17.441073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440959 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-data\") pod \"b39d793a-45dc-4682-b068-7666d06bb566\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " Oct 30 01:19:17.441073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.440990 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-credential-socket\") pod \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " Oct 30 01:19:17.441073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.441057 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-credential-socket\") pod \"b39d793a-45dc-4682-b068-7666d06bb566\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " Oct 30 01:19:17.441277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.441101 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-certs\") pod \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " Oct 30 01:19:17.441277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.441156 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-kube-api-access-qqmf8\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-kube-api-access-qqmf8\") pod \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " Oct 30 01:19:17.441277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.441192 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b39d793a-45dc-4682-b068-7666d06bb566-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d793a-45dc-4682-b068-7666d06bb566-istio-podinfo\") pod \"b39d793a-45dc-4682-b068-7666d06bb566\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " Oct 30 01:19:17.441277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.441259 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-token\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-token\") pod \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " Oct 30 01:19:17.441500 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.441291 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-kube-api-access-bfvz6\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-kube-api-access-bfvz6\") pod \"b39d793a-45dc-4682-b068-7666d06bb566\" (UID: \"b39d793a-45dc-4682-b068-7666d06bb566\") " Oct 30 01:19:17.441500 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.441346 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-podinfo\") pod \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\" (UID: \"ee1086e2-a267-44c9-b5a8-f94c1aa59385\") " Oct 30 01:19:17.443018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.442889 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39d793a-45dc-4682-b068-7666d06bb566-istiod-ca-cert" pod "b39d793a-45dc-4682-b068-7666d06bb566" (UID: "b39d793a-45dc-4682-b068-7666d06bb566"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:17.444764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.443201 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-socket" pod "ee1086e2-a267-44c9-b5a8-f94c1aa59385" (UID: "ee1086e2-a267-44c9-b5a8-f94c1aa59385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.444764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.443544 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-socket" pod "b39d793a-45dc-4682-b068-7666d06bb566" (UID: "b39d793a-45dc-4682-b068-7666d06bb566"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.444764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.443588 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istiod-ca-cert" pod "ee1086e2-a267-44c9-b5a8-f94c1aa59385" (UID: "ee1086e2-a267-44c9-b5a8-f94c1aa59385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:17.444764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.444037 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-certs" pod "b39d793a-45dc-4682-b068-7666d06bb566" (UID: "b39d793a-45dc-4682-b068-7666d06bb566"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.444764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.444069 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-certs" pod "ee1086e2-a267-44c9-b5a8-f94c1aa59385" (UID: "ee1086e2-a267-44c9-b5a8-f94c1aa59385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.444764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.444253 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-data" pod "b39d793a-45dc-4682-b068-7666d06bb566" (UID: "b39d793a-45dc-4682-b068-7666d06bb566"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.444764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.444406 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-data" pod "ee1086e2-a267-44c9-b5a8-f94c1aa59385" (UID: "ee1086e2-a267-44c9-b5a8-f94c1aa59385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.444764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.444419 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-credential-socket" pod "ee1086e2-a267-44c9-b5a8-f94c1aa59385" (UID: "ee1086e2-a267-44c9-b5a8-f94c1aa59385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.444764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.444579 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-credential-socket" pod "b39d793a-45dc-4682-b068-7666d06bb566" (UID: "b39d793a-45dc-4682-b068-7666d06bb566"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.445380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.445166 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-podinfo" pod "ee1086e2-a267-44c9-b5a8-f94c1aa59385" (UID: "ee1086e2-a267-44c9-b5a8-f94c1aa59385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:17.447971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.447936 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-kube-api-access-qqmf8" pod "ee1086e2-a267-44c9-b5a8-f94c1aa59385" (UID: "ee1086e2-a267-44c9-b5a8-f94c1aa59385"). InnerVolumeSpecName "kube-api-access-qqmf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:17.449664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.449631 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-kube-api-access-bfvz6" pod "b39d793a-45dc-4682-b068-7666d06bb566" (UID: "b39d793a-45dc-4682-b068-7666d06bb566"). InnerVolumeSpecName "kube-api-access-bfvz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:17.450132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.450105 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b39d793a-45dc-4682-b068-7666d06bb566-istio-podinfo" pod "b39d793a-45dc-4682-b068-7666d06bb566" (UID: "b39d793a-45dc-4682-b068-7666d06bb566"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:17.450348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.450320 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-istio-token" pod "b39d793a-45dc-4682-b068-7666d06bb566" (UID: "b39d793a-45dc-4682-b068-7666d06bb566"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:17.451633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.451596 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-token" pod "ee1086e2-a267-44c9-b5a8-f94c1aa59385" (UID: "ee1086e2-a267-44c9-b5a8-f94c1aa59385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:17.452937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.452908 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-envoy" pod "b39d793a-45dc-4682-b068-7666d06bb566" (UID: "b39d793a-45dc-4682-b068-7666d06bb566"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.456900 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.456861 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-envoy" pod "ee1086e2-a267-44c9-b5a8-f94c1aa59385" (UID: "ee1086e2-a267-44c9-b5a8-f94c1aa59385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544664 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544726 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544745 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544758 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544771 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544784 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d793a-45dc-4682-b068-7666d06bb566-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544797 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544809 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544822 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544837 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544851 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmf8\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-kube-api-access-qqmf8\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544867 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d793a-45dc-4682-b068-7666d06bb566-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544880 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544894 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfvz6\" (UniqueName: \"kubernetes.io/projected/b39d793a-45dc-4682-b068-7666d06bb566-kube-api-access-bfvz6\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544906 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544920 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544931 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1086e2-a267-44c9-b5a8-f94c1aa59385-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.545026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.544944 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d793a-45dc-4682-b068-7666d06bb566-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.822191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.822160 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m"] Oct 30 01:19:17.824459 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:17.824430 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cf5570_1952_4038_af67_dded46578a90.slice/crio-dd09f9eda20b040e5ab83d0990e761704505f42540f4792e49c883fbdac13584 WatchSource:0}: Error finding container dd09f9eda20b040e5ab83d0990e761704505f42540f4792e49c883fbdac13584: Status 404 returned error can't find the container with id dd09f9eda20b040e5ab83d0990e761704505f42540f4792e49c883fbdac13584 Oct 30 01:19:17.828145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.828099 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:19:17.828266 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.828181 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:19:17.828266 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.828212 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:19:17.872946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.872913 4517 generic.go:356] "Generic (PLEG): container finished" podID="3f823884-3933-465d-a308-c27be99ef1a3" containerID="d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a" exitCode=0 Oct 30 01:19:17.873123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.872977 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" Oct 30 01:19:17.873123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.873002 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" event={"ID":"3f823884-3933-465d-a308-c27be99ef1a3","Type":"ContainerDied","Data":"d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a"} Oct 30 01:19:17.873123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.873072 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b" event={"ID":"3f823884-3933-465d-a308-c27be99ef1a3","Type":"ContainerDied","Data":"cbb844b37a329882230d6065e0b8680f7e4bf6559780c2e026282ba9203e1244"} Oct 30 01:19:17.873123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.873104 4517 scope.go:122] "RemoveContainer" containerID="d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a" Oct 30 01:19:17.883669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.883637 4517 generic.go:356] "Generic (PLEG): container finished" podID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerID="29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f" exitCode=0 Oct 30 01:19:17.884288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.883766 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" event={"ID":"ee1086e2-a267-44c9-b5a8-f94c1aa59385","Type":"ContainerDied","Data":"29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f"} Oct 30 01:19:17.884288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.883842 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" event={"ID":"ee1086e2-a267-44c9-b5a8-f94c1aa59385","Type":"ContainerDied","Data":"e0daaa1146bfe360f555abbf91a5577d09528298cab51bb62c7c0a81611c9d61"} Oct 30 01:19:17.884288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.883929 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7" Oct 30 01:19:17.888428 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.887782 4517 scope.go:122] "RemoveContainer" containerID="d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a" Oct 30 01:19:17.889654 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.889617 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" event={"ID":"35cf5570-1952-4038-af67-dded46578a90","Type":"ContainerStarted","Data":"dd09f9eda20b040e5ab83d0990e761704505f42540f4792e49c883fbdac13584"} Oct 30 01:19:17.890007 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:17.889762 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a\": container with ID starting with d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a not found: ID does not exist" containerID="d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a" Oct 30 01:19:17.890108 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.890025 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a"} err="failed to get container status \"d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a\": rpc error: code = NotFound desc = could not find container \"d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a\": container with ID starting with d560977a8eaae5754c91275802df52d75a1ea3faa47eb6f79ce2e2472820d95a not found: ID does not exist" Oct 30 01:19:17.890108 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.890049 4517 scope.go:122] "RemoveContainer" containerID="29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f" Oct 30 01:19:17.903440 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.903332 4517 generic.go:356] "Generic (PLEG): container finished" podID="b39d793a-45dc-4682-b068-7666d06bb566" containerID="a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438" exitCode=0 Oct 30 01:19:17.903684 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.903498 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" Oct 30 01:19:17.903684 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.903512 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb" Oct 30 01:19:17.903684 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.903541 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" event={"ID":"b39d793a-45dc-4682-b068-7666d06bb566","Type":"ContainerDied","Data":"a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438"} Oct 30 01:19:17.903684 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.903605 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr" event={"ID":"b39d793a-45dc-4682-b068-7666d06bb566","Type":"ContainerDied","Data":"51582df759613a6084077759f01b2d618fe7260e8a7ae857a174450fed82caf6"} Oct 30 01:19:17.933902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.933874 4517 scope.go:122] "RemoveContainer" containerID="29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f" Oct 30 01:19:17.934336 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:17.934303 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f\": container with ID starting with 29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f not found: ID does not exist" containerID="29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f" Oct 30 01:19:17.934451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.934350 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f"} err="failed to get container status \"29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f\": rpc error: code = NotFound desc = could not find container \"29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f\": container with ID starting with 29c5d516e0f1253824a85f9d9bfb8e70065f7ea48e87fb6943d6fce778e1021f not found: ID does not exist" Oct 30 01:19:17.934451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.934379 4517 scope.go:122] "RemoveContainer" containerID="a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438" Oct 30 01:19:17.946776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.946743 4517 scope.go:122] "RemoveContainer" containerID="a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438" Oct 30 01:19:17.947235 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:17.947206 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438\": container with ID starting with a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438 not found: ID does not exist" containerID="a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438" Oct 30 01:19:17.947305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.947251 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438"} err="failed to get container status \"a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438\": rpc error: code = NotFound desc = could not find container \"a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438\": container with ID starting with a64849abe2bbd23f6fe353210a04c91f27b7d954c0f5f70e507a570def97c438 not found: ID does not exist" Oct 30 01:19:17.958487 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.958439 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b"] Oct 30 01:19:17.967510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:17.967472 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-vgh7b"] Oct 30 01:19:18.009928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.009888 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr"] Oct 30 01:19:18.017465 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.017429 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-kjtfr"] Oct 30 01:19:18.045545 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.045504 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb"] Oct 30 01:19:18.051420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.051385 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-jphbb"] Oct 30 01:19:18.079765 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.079644 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7"] Oct 30 01:19:18.086041 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.085997 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-4m7f7"] Oct 30 01:19:18.222312 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.222279 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:18.238121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.238082 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336252d7-b778-490b-93e8-7f2c4ca5db60" path="/var/lib/kubelet/pods/336252d7-b778-490b-93e8-7f2c4ca5db60/volumes" Oct 30 01:19:18.238689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.238673 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f823884-3933-465d-a308-c27be99ef1a3" path="/var/lib/kubelet/pods/3f823884-3933-465d-a308-c27be99ef1a3/volumes" Oct 30 01:19:18.239197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.239183 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39d793a-45dc-4682-b068-7666d06bb566" path="/var/lib/kubelet/pods/b39d793a-45dc-4682-b068-7666d06bb566/volumes" Oct 30 01:19:18.240044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.240027 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" path="/var/lib/kubelet/pods/ee1086e2-a267-44c9-b5a8-f94c1aa59385/volumes" Oct 30 01:19:18.917350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.917306 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" event={"ID":"35cf5570-1952-4038-af67-dded46578a90","Type":"ContainerStarted","Data":"eb2158fbbbb8d4b5224b7904584a5a9e03eb1c35239ed3e6f4aad2aa1111ff4a"} Oct 30 01:19:18.962792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.962666 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" podStartSLOduration=124.96264089 podStartE2EDuration="2m4.96264089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:18.961042862 +0000 UTC m=+2696.873170046" watchObservedRunningTime="2026-10-30 01:19:18.96264089 +0000 UTC m=+2696.874768033" Oct 30 01:19:18.995300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.995262 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x"] Oct 30 01:19:18.996085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.996063 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" Oct 30 01:19:18.996085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.996086 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" Oct 30 01:19:18.996292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.996142 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" Oct 30 01:19:18.996292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.996150 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" Oct 30 01:19:18.996292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.996208 4517 cpu_manager.go:414] "containerMap: removing container" podUID="3f823884-3933-465d-a308-c27be99ef1a3" containerName="istio-proxy" Oct 30 01:19:18.996292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.996216 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f823884-3933-465d-a308-c27be99ef1a3" containerName="istio-proxy" Oct 30 01:19:18.996484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.996398 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f823884-3933-465d-a308-c27be99ef1a3" containerName="istio-proxy" Oct 30 01:19:18.996484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.996418 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1086e2-a267-44c9-b5a8-f94c1aa59385" containerName="istio-proxy" Oct 30 01:19:18.996484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:18.996430 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39d793a-45dc-4682-b068-7666d06bb566" containerName="istio-proxy" Oct 30 01:19:19.000190 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.000163 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.008385 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.008353 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:19.021471 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.021417 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x"] Oct 30 01:19:19.062046 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.062001 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:19:19.062284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.062079 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:19:19.134868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.134510 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.134868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.134691 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7db6v\" (UniqueName: \"kubernetes.io/projected/f07ce999-326b-4fcd-998b-abf089f12248-kube-api-access-7db6v\") pod \"maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.134868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.134857 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.237985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.237881 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db6v\" (UniqueName: \"kubernetes.io/projected/f07ce999-326b-4fcd-998b-abf089f12248-kube-api-access-7db6v\") pod \"maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.238317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.238013 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.238317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.238277 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.239253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.239227 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.241105 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.241077 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.260839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.260800 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7db6v\" (UniqueName: \"kubernetes.io/projected/f07ce999-326b-4fcd-998b-abf089f12248-kube-api-access-7db6v\") pod \"maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.323913 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.323643 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:19.443730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.441369 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:19.450638 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.450600 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:19.732077 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.732050 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x"] Oct 30 01:19:19.734444 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:19.734409 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07ce999_326b_4fcd_998b_abf089f12248.slice/crio-a0e9b06bc1466d486b6f9efbdc8bce166817f8829be14cda509f85806b832ad2 WatchSource:0}: Error finding container a0e9b06bc1466d486b6f9efbdc8bce166817f8829be14cda509f85806b832ad2: Status 404 returned error can't find the container with id a0e9b06bc1466d486b6f9efbdc8bce166817f8829be14cda509f85806b832ad2 Oct 30 01:19:19.928402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.928360 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" event={"ID":"f07ce999-326b-4fcd-998b-abf089f12248","Type":"ContainerStarted","Data":"a0e9b06bc1466d486b6f9efbdc8bce166817f8829be14cda509f85806b832ad2"} Oct 30 01:19:19.928871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.928807 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:19.930879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:19.930853 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:19:20.933319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:20.933284 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:20.939498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:20.939467 4517 generic.go:356] "Generic (PLEG): container finished" podID="f07ce999-326b-4fcd-998b-abf089f12248" containerID="276d3bda6ccc0eb6b4de478b9a5f6f4b3a9844c0b95227208880e4fb184a037e" exitCode=0 Oct 30 01:19:20.939688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:20.939556 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" event={"ID":"f07ce999-326b-4fcd-998b-abf089f12248","Type":"ContainerDied","Data":"276d3bda6ccc0eb6b4de478b9a5f6f4b3a9844c0b95227208880e4fb184a037e"} Oct 30 01:19:21.278634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:21.278566 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:21.390186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:21.390149 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:22.149613 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.149581 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:22.312205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.312169 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07ce999-326b-4fcd-998b-abf089f12248-kube-api-access-7db6v\" (UniqueName: \"kubernetes.io/projected/f07ce999-326b-4fcd-998b-abf089f12248-kube-api-access-7db6v\") pod \"f07ce999-326b-4fcd-998b-abf089f12248\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " Oct 30 01:19:22.312418 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.312293 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-xks-ca\") pod \"f07ce999-326b-4fcd-998b-abf089f12248\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " Oct 30 01:19:22.312603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.312576 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-openshift-ca\") pod \"f07ce999-326b-4fcd-998b-abf089f12248\" (UID: \"f07ce999-326b-4fcd-998b-abf089f12248\") " Oct 30 01:19:22.313050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.313021 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-openshift-ca" pod "f07ce999-326b-4fcd-998b-abf089f12248" (UID: "f07ce999-326b-4fcd-998b-abf089f12248"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:22.313558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.313536 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:22.315565 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.315533 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07ce999-326b-4fcd-998b-abf089f12248-kube-api-access-7db6v" pod "f07ce999-326b-4fcd-998b-abf089f12248" (UID: "f07ce999-326b-4fcd-998b-abf089f12248"). InnerVolumeSpecName "kube-api-access-7db6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:22.315912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.315872 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-xks-ca" pod "f07ce999-326b-4fcd-998b-abf089f12248" (UID: "f07ce999-326b-4fcd-998b-abf089f12248"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:22.338203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.338147 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj"] Oct 30 01:19:22.345114 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.345083 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf"] Oct 30 01:19:22.415619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.415222 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7db6v\" (UniqueName: \"kubernetes.io/projected/f07ce999-326b-4fcd-998b-abf089f12248-kube-api-access-7db6v\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:22.415619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.415264 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07ce999-326b-4fcd-998b-abf089f12248-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:22.681161 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.681074 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:22.967163 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.967056 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" Oct 30 01:19:22.967373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.967173 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x" event={"ID":"f07ce999-326b-4fcd-998b-abf089f12248","Type":"ContainerDied","Data":"a0e9b06bc1466d486b6f9efbdc8bce166817f8829be14cda509f85806b832ad2"} Oct 30 01:19:22.967373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:22.967239 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e9b06bc1466d486b6f9efbdc8bce166817f8829be14cda509f85806b832ad2" Oct 30 01:19:23.035565 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:23.034699 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:23.682736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:23.681400 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:23.804782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:23.804749 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:24.304448 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.304411 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt"] Oct 30 01:19:24.305803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.305774 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f07ce999-326b-4fcd-998b-abf089f12248" containerName="revoke-keys" Oct 30 01:19:24.305996 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.305981 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07ce999-326b-4fcd-998b-abf089f12248" containerName="revoke-keys" Oct 30 01:19:24.306356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.306336 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07ce999-326b-4fcd-998b-abf089f12248" containerName="revoke-keys" Oct 30 01:19:24.313086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.313051 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:24.340039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.340005 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5f99e7\"" Oct 30 01:19:24.340700 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.340340 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5f99e7-dockercfg-x7kdx\"" Oct 30 01:19:24.341778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.341036 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt"] Oct 30 01:19:24.498369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.498319 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqkp\" (UniqueName: \"kubernetes.io/projected/b46f7744-bbd1-4bdd-a247-88f0135a1285-kube-api-access-dlqkp\") pod \"payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt\" (UID: \"b46f7744-bbd1-4bdd-a247-88f0135a1285\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:24.498609 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.498569 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46f7744-bbd1-4bdd-a247-88f0135a1285-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt\" (UID: \"b46f7744-bbd1-4bdd-a247-88f0135a1285\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:24.600643 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.600527 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqkp\" (UniqueName: \"kubernetes.io/projected/b46f7744-bbd1-4bdd-a247-88f0135a1285-kube-api-access-dlqkp\") pod \"payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt\" (UID: \"b46f7744-bbd1-4bdd-a247-88f0135a1285\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:24.600907 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.600701 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46f7744-bbd1-4bdd-a247-88f0135a1285-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt\" (UID: \"b46f7744-bbd1-4bdd-a247-88f0135a1285\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:24.601832 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.601796 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46f7744-bbd1-4bdd-a247-88f0135a1285-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt\" (UID: \"b46f7744-bbd1-4bdd-a247-88f0135a1285\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:24.625335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.625302 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqkp\" (UniqueName: \"kubernetes.io/projected/b46f7744-bbd1-4bdd-a247-88f0135a1285-kube-api-access-dlqkp\") pod \"payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt\" (UID: \"b46f7744-bbd1-4bdd-a247-88f0135a1285\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:24.644289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.644243 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:24.964094 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:24.963930 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt"] Oct 30 01:19:24.966058 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:24.966010 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46f7744_bbd1_4bdd_a247_88f0135a1285.slice/crio-5e8f384ce091fba016264a1e915b2a62227b58bf3c79819aa541be7a0726cfc9 WatchSource:0}: Error finding container 5e8f384ce091fba016264a1e915b2a62227b58bf3c79819aa541be7a0726cfc9: Status 404 returned error can't find the container with id 5e8f384ce091fba016264a1e915b2a62227b58bf3c79819aa541be7a0726cfc9 Oct 30 01:19:25.299805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:25.299757 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:25.445112 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:25.445074 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" event={"ID":"b46f7744-bbd1-4bdd-a247-88f0135a1285","Type":"ContainerStarted","Data":"ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb"} Oct 30 01:19:25.445329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:25.445130 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" event={"ID":"b46f7744-bbd1-4bdd-a247-88f0135a1285","Type":"ContainerStarted","Data":"5e8f384ce091fba016264a1e915b2a62227b58bf3c79819aa541be7a0726cfc9"} Oct 30 01:19:25.446216 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:25.446196 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:25.483459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:25.483388 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" podStartSLOduration=2.4833674390000002 podStartE2EDuration="2.483367439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:25.481934129 +0000 UTC m=+2703.394061271" watchObservedRunningTime="2026-10-30 01:19:25.483367439 +0000 UTC m=+2703.395494580" Oct 30 01:19:26.173398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:26.173364 4517 scope.go:122] "RemoveContainer" containerID="9b64fe4c73cdfeb5dfde29a6bce533cbe653428d8e0ea9a2ba12dda6e2836f49" Oct 30 01:19:26.185006 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:26.184972 4517 scope.go:122] "RemoveContainer" containerID="3b106acf8d288431595661551e1af87894e3b5ee8aabda61b5ce157354c4bd22" Oct 30 01:19:26.196396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:26.196366 4517 scope.go:122] "RemoveContainer" containerID="dd28435aa7c5c445eed4608fef943e180d331cf88688940ca9122ada7d7d13e4" Oct 30 01:19:26.207863 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:26.207837 4517 scope.go:122] "RemoveContainer" containerID="9dadb2692b2882f138f67f567644f4f7adcb606a79c00c303b28986290adb7d6" Oct 30 01:19:27.025170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:27.025125 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:27.339509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:27.339460 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" podUID="3c7ea040-73a1-4179-8b73-4ad5109192a8" containerName="payload-processing" containerID="cri-o://7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5" gracePeriod=25 Oct 30 01:19:27.345670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:27.345628 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" podUID="844c10a2-f543-4898-b69f-b207052e0d8d" containerName="payload-pre-processing" containerID="cri-o://31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50" gracePeriod=25 Oct 30 01:19:29.019951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:29.019912 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:30.660573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:30.660529 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:30.808195 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:30.808142 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:19:32.221317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:32.221272 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:32.844410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:32.844371 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95"] Oct 30 01:19:32.851060 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:32.851020 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:32.870620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:32.870568 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95"] Oct 30 01:19:32.897778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:32.896627 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:19:32.897778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:32.896951 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:19:32.997829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:32.997789 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krqc\" (UniqueName: \"kubernetes.io/projected/923b47d8-5e17-4531-8114-12ccbc48e150-kube-api-access-8krqc\") pod \"maas-api-revoke-sub-45ebae79-grm95\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:32.998044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:32.997952 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45ebae79-grm95\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:32.998044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:32.998008 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45ebae79-grm95\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:33.100239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.100121 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krqc\" (UniqueName: \"kubernetes.io/projected/923b47d8-5e17-4531-8114-12ccbc48e150-kube-api-access-8krqc\") pod \"maas-api-revoke-sub-45ebae79-grm95\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:33.100451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.100243 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45ebae79-grm95\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:33.100451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.100289 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45ebae79-grm95\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:33.101252 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.101214 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45ebae79-grm95\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:33.103023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.102996 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45ebae79-grm95\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:33.122599 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.122563 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krqc\" (UniqueName: \"kubernetes.io/projected/923b47d8-5e17-4531-8114-12ccbc48e150-kube-api-access-8krqc\") pod \"maas-api-revoke-sub-45ebae79-grm95\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:33.170171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.170131 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:33.487498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.487459 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95"] Oct 30 01:19:33.490377 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:33.490333 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923b47d8_5e17_4531_8114_12ccbc48e150.slice/crio-58ff861a6a302ff4bf0ec470df29b30385acbe1b3e9c53c5ddb014760a3f6fcc WatchSource:0}: Error finding container 58ff861a6a302ff4bf0ec470df29b30385acbe1b3e9c53c5ddb014760a3f6fcc: Status 404 returned error can't find the container with id 58ff861a6a302ff4bf0ec470df29b30385acbe1b3e9c53c5ddb014760a3f6fcc Oct 30 01:19:33.551528 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.551484 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" event={"ID":"923b47d8-5e17-4531-8114-12ccbc48e150","Type":"ContainerStarted","Data":"58ff861a6a302ff4bf0ec470df29b30385acbe1b3e9c53c5ddb014760a3f6fcc"} Oct 30 01:19:33.913864 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:33.913810 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:34.367181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:34.367138 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:34.562403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:34.562360 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" event={"ID":"923b47d8-5e17-4531-8114-12ccbc48e150","Type":"ContainerStarted","Data":"bccefeeeb348be7012d832f6be4db07034172b8d81195e499b9b851f31de76f6"} Oct 30 01:19:34.601903 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:34.601845 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" podStartSLOduration=2.601828374 podStartE2EDuration="2.601828374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:34.599194637 +0000 UTC m=+2712.511321800" watchObservedRunningTime="2026-10-30 01:19:34.601828374 +0000 UTC m=+2712.513955514" Oct 30 01:19:34.623676 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:34.623587 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:34.945997 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:34.942530 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:35.186552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:35.170820 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z"] Oct 30 01:19:35.186552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:35.183657 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:35.532931 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:35.524359 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20155e60-nxg2z"] Oct 30 01:19:35.910175 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:35.910135 4517 generic.go:356] "Generic (PLEG): container finished" podID="923b47d8-5e17-4531-8114-12ccbc48e150" containerID="bccefeeeb348be7012d832f6be4db07034172b8d81195e499b9b851f31de76f6" exitCode=0 Oct 30 01:19:35.911324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:35.910916 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" event={"ID":"923b47d8-5e17-4531-8114-12ccbc48e150","Type":"ContainerDied","Data":"bccefeeeb348be7012d832f6be4db07034172b8d81195e499b9b851f31de76f6"} Oct 30 01:19:35.918849 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:35.918799 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:35.924562 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:35.924482 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:35.972735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:35.964845 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:36.081270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:36.081193 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:36.196262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:36.196129 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:36.240984 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:36.240945 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3770123-5dcb-4bb5-b671-34010ea73246" path="/var/lib/kubelet/pods/d3770123-5dcb-4bb5-b671-34010ea73246/volumes" Oct 30 01:19:36.334468 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:36.334430 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.162139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.162099 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:37.336530 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.336488 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" podUID="844c10a2-f543-4898-b69f-b207052e0d8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:19:37.336817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.336792 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" podUID="3c7ea040-73a1-4179-8b73-4ad5109192a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:19:37.363242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.363197 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-xks-ca\") pod \"923b47d8-5e17-4531-8114-12ccbc48e150\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " Oct 30 01:19:37.363463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.363297 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-openshift-ca\") pod \"923b47d8-5e17-4531-8114-12ccbc48e150\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " Oct 30 01:19:37.363898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.363867 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/923b47d8-5e17-4531-8114-12ccbc48e150-kube-api-access-8krqc\" (UniqueName: \"kubernetes.io/projected/923b47d8-5e17-4531-8114-12ccbc48e150-kube-api-access-8krqc\") pod \"923b47d8-5e17-4531-8114-12ccbc48e150\" (UID: \"923b47d8-5e17-4531-8114-12ccbc48e150\") " Oct 30 01:19:37.364275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.364244 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-openshift-ca" pod "923b47d8-5e17-4531-8114-12ccbc48e150" (UID: "923b47d8-5e17-4531-8114-12ccbc48e150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:37.365370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.365057 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.373329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.373277 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923b47d8-5e17-4531-8114-12ccbc48e150-kube-api-access-8krqc" pod "923b47d8-5e17-4531-8114-12ccbc48e150" (UID: "923b47d8-5e17-4531-8114-12ccbc48e150"). InnerVolumeSpecName "kube-api-access-8krqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:37.407566 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.407379 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-xks-ca" pod "923b47d8-5e17-4531-8114-12ccbc48e150" (UID: "923b47d8-5e17-4531-8114-12ccbc48e150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:37.468139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.468088 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8krqc\" (UniqueName: \"kubernetes.io/projected/923b47d8-5e17-4531-8114-12ccbc48e150-kube-api-access-8krqc\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.468395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.468378 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923b47d8-5e17-4531-8114-12ccbc48e150-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.559494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.557943 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:19:37.903526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:37.903476 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:38.061731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:38.057742 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" Oct 30 01:19:38.061731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:38.057794 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95" event={"ID":"923b47d8-5e17-4531-8114-12ccbc48e150","Type":"ContainerDied","Data":"58ff861a6a302ff4bf0ec470df29b30385acbe1b3e9c53c5ddb014760a3f6fcc"} Oct 30 01:19:38.061731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:38.057956 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58ff861a6a302ff4bf0ec470df29b30385acbe1b3e9c53c5ddb014760a3f6fcc" Oct 30 01:19:39.022572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:39.022532 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:39.150960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:39.150920 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:39.298014 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:39.297980 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:39.448818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:39.448777 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:40.001735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:39.998521 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:40.696689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.696650 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:40.818548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.818505 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:40.885452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.885413 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w"] Oct 30 01:19:40.886186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.886170 4517 cpu_manager.go:414] "containerMap: removing container" podUID="923b47d8-5e17-4531-8114-12ccbc48e150" containerName="revoke-keys" Oct 30 01:19:40.886267 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.886187 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="923b47d8-5e17-4531-8114-12ccbc48e150" containerName="revoke-keys" Oct 30 01:19:40.886303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.886290 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="923b47d8-5e17-4531-8114-12ccbc48e150" containerName="revoke-keys" Oct 30 01:19:40.890158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.890124 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:40.903939 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.903906 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6cba49\"" Oct 30 01:19:40.919068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.919033 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6cba49-dockercfg-npxcn\"" Oct 30 01:19:40.920398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.920368 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w"] Oct 30 01:19:40.977447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.977349 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba845c42-6120-4c91-b619-5d03dd41393e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w\" (UID: \"ba845c42-6120-4c91-b619-5d03dd41393e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:40.977751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:40.977680 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwjv\" (UniqueName: \"kubernetes.io/projected/ba845c42-6120-4c91-b619-5d03dd41393e-kube-api-access-9kwjv\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w\" (UID: \"ba845c42-6120-4c91-b619-5d03dd41393e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:41.125976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.114418 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba845c42-6120-4c91-b619-5d03dd41393e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w\" (UID: \"ba845c42-6120-4c91-b619-5d03dd41393e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:41.125976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.114492 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwjv\" (UniqueName: \"kubernetes.io/projected/ba845c42-6120-4c91-b619-5d03dd41393e-kube-api-access-9kwjv\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w\" (UID: \"ba845c42-6120-4c91-b619-5d03dd41393e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:41.125976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.115905 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba845c42-6120-4c91-b619-5d03dd41393e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w\" (UID: \"ba845c42-6120-4c91-b619-5d03dd41393e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:41.385106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.385065 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt"] Oct 30 01:19:41.393867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.393834 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:41.403065 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.403030 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwjv\" (UniqueName: \"kubernetes.io/projected/ba845c42-6120-4c91-b619-5d03dd41393e-kube-api-access-9kwjv\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w\" (UID: \"ba845c42-6120-4c91-b619-5d03dd41393e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:41.411584 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.411529 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt"] Oct 30 01:19:41.474912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.474870 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c485ddbf-39d0-4c2f-b45c-4900a9775566-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt\" (UID: \"c485ddbf-39d0-4c2f-b45c-4900a9775566\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:41.475180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.475148 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5kjq\" (UniqueName: \"kubernetes.io/projected/c485ddbf-39d0-4c2f-b45c-4900a9775566-kube-api-access-b5kjq\") pod \"payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt\" (UID: \"c485ddbf-39d0-4c2f-b45c-4900a9775566\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:41.507268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.507221 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:41.533913 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.533879 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc"] Oct 30 01:19:41.534214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.534183 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" containerID="cri-o://1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038" gracePeriod=30 Oct 30 01:19:41.585205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.582412 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c485ddbf-39d0-4c2f-b45c-4900a9775566-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt\" (UID: \"c485ddbf-39d0-4c2f-b45c-4900a9775566\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:41.585205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.582637 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5kjq\" (UniqueName: \"kubernetes.io/projected/c485ddbf-39d0-4c2f-b45c-4900a9775566-kube-api-access-b5kjq\") pod \"payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt\" (UID: \"c485ddbf-39d0-4c2f-b45c-4900a9775566\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:41.585205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.584410 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c485ddbf-39d0-4c2f-b45c-4900a9775566-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt\" (UID: \"c485ddbf-39d0-4c2f-b45c-4900a9775566\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:41.609495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.607789 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5kjq\" (UniqueName: \"kubernetes.io/projected/c485ddbf-39d0-4c2f-b45c-4900a9775566-kube-api-access-b5kjq\") pod \"payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt\" (UID: \"c485ddbf-39d0-4c2f-b45c-4900a9775566\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:41.727284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.727191 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:41.840623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.840589 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w"] Oct 30 01:19:41.841916 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:41.841875 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba845c42_6120_4c91_b619_5d03dd41393e.slice/crio-a03c811bf821dd8c74e8410f4ba4fef878a6267005ac41e0cb9bfc425a08be09 WatchSource:0}: Error finding container a03c811bf821dd8c74e8410f4ba4fef878a6267005ac41e0cb9bfc425a08be09: Status 404 returned error can't find the container with id a03c811bf821dd8c74e8410f4ba4fef878a6267005ac41e0cb9bfc425a08be09 Oct 30 01:19:41.943095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:41.943029 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:42.087360 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:42.087326 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc485ddbf_39d0_4c2f_b45c_4900a9775566.slice/crio-155ccadb9d8ba7f09215e2553be009e07305105acd14b14f939cd6f85a9f3ce8 WatchSource:0}: Error finding container 155ccadb9d8ba7f09215e2553be009e07305105acd14b14f939cd6f85a9f3ce8: Status 404 returned error can't find the container with id 155ccadb9d8ba7f09215e2553be009e07305105acd14b14f939cd6f85a9f3ce8 Oct 30 01:19:42.090171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.090140 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt"] Oct 30 01:19:42.330373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.330335 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" event={"ID":"ba845c42-6120-4c91-b619-5d03dd41393e","Type":"ContainerStarted","Data":"5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b"} Oct 30 01:19:42.330601 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.330402 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" event={"ID":"ba845c42-6120-4c91-b619-5d03dd41393e","Type":"ContainerStarted","Data":"a03c811bf821dd8c74e8410f4ba4fef878a6267005ac41e0cb9bfc425a08be09"} Oct 30 01:19:42.330601 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.330460 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:42.347018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.346900 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" event={"ID":"c485ddbf-39d0-4c2f-b45c-4900a9775566","Type":"ContainerStarted","Data":"33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac"} Oct 30 01:19:42.347018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.347016 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" event={"ID":"c485ddbf-39d0-4c2f-b45c-4900a9775566","Type":"ContainerStarted","Data":"155ccadb9d8ba7f09215e2553be009e07305105acd14b14f939cd6f85a9f3ce8"} Oct 30 01:19:42.347287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.347143 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:42.398657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.398603 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:42.412412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.412337 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" podStartSLOduration=2.41231415 podStartE2EDuration="2.41231415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:42.386286495 +0000 UTC m=+2720.298413668" watchObservedRunningTime="2026-10-30 01:19:42.41231415 +0000 UTC m=+2720.324441290" Oct 30 01:19:42.444673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.444612 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" podStartSLOduration=2.444594318 podStartE2EDuration="2.444594318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:42.439850344 +0000 UTC m=+2720.351977496" watchObservedRunningTime="2026-10-30 01:19:42.444594318 +0000 UTC m=+2720.356721452" Oct 30 01:19:42.519969 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.519928 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4"] Oct 30 01:19:42.524804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.524778 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:42.537902 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.537857 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fc3cb2\"" Oct 30 01:19:42.538116 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.538071 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fc3cb2-dockercfg-hzqv2\"" Oct 30 01:19:42.543884 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.543854 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4"] Oct 30 01:19:42.565692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.565633 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:42.633480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.626940 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptt2v\" (UniqueName: \"kubernetes.io/projected/01bd6e21-5e90-4063-956d-09f0b7523bd2-kube-api-access-ptt2v\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4\" (UID: \"01bd6e21-5e90-4063-956d-09f0b7523bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:42.633480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.627095 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bd6e21-5e90-4063-956d-09f0b7523bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4\" (UID: \"01bd6e21-5e90-4063-956d-09f0b7523bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:42.691400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.679082 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp"] Oct 30 01:19:42.740813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.731144 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptt2v\" (UniqueName: \"kubernetes.io/projected/01bd6e21-5e90-4063-956d-09f0b7523bd2-kube-api-access-ptt2v\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4\" (UID: \"01bd6e21-5e90-4063-956d-09f0b7523bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:42.744277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.742973 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bd6e21-5e90-4063-956d-09f0b7523bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4\" (UID: \"01bd6e21-5e90-4063-956d-09f0b7523bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:42.744277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.744212 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bd6e21-5e90-4063-956d-09f0b7523bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4\" (UID: \"01bd6e21-5e90-4063-956d-09f0b7523bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:42.758127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.758085 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:42.797738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.797172 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp"] Oct 30 01:19:42.860324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.860273 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xppcq\" (UniqueName: \"kubernetes.io/projected/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-kube-api-access-xppcq\") pod \"payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp\" (UID: \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:42.860504 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.860486 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp\" (UID: \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:43.006783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:42.982914 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xppcq\" (UniqueName: \"kubernetes.io/projected/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-kube-api-access-xppcq\") pod \"payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp\" (UID: \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:43.013733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.013673 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp\" (UID: \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:43.015167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.004776 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptt2v\" (UniqueName: \"kubernetes.io/projected/01bd6e21-5e90-4063-956d-09f0b7523bd2-kube-api-access-ptt2v\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4\" (UID: \"01bd6e21-5e90-4063-956d-09f0b7523bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:43.015387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.014830 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp\" (UID: \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:43.016915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.016887 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c"] Oct 30 01:19:43.027440 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.027407 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xppcq\" (UniqueName: \"kubernetes.io/projected/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-kube-api-access-xppcq\") pod \"payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp\" (UID: \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:43.102588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.102551 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:43.142453 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.142411 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:43.225282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.223153 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:43.320185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.319982 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh"] Oct 30 01:19:43.333495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.331224 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.350504 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.347903 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:43.374079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.374045 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh"] Oct 30 01:19:43.409286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.408906 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:19:43.409286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.409010 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:19:43.450064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.448319 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ddf68f-rgfwh\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.450064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.448490 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ddf68f-rgfwh\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.450064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.448577 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbnrv\" (UniqueName: \"kubernetes.io/projected/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-kube-api-access-gbnrv\") pod \"maas-api-revoke-sub-08ddf68f-rgfwh\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.535871 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:43.535831 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01bd6e21_5e90_4063_956d_09f0b7523bd2.slice/crio-cbfc1e0211482a9ee6b790d6e9a8ece46d26d8c8a1c78d0ceac51c22997d7434 WatchSource:0}: Error finding container cbfc1e0211482a9ee6b790d6e9a8ece46d26d8c8a1c78d0ceac51c22997d7434: Status 404 returned error can't find the container with id cbfc1e0211482a9ee6b790d6e9a8ece46d26d8c8a1c78d0ceac51c22997d7434 Oct 30 01:19:43.537188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.537153 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4"] Oct 30 01:19:43.550532 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.550486 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ddf68f-rgfwh\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.550736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.550605 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbnrv\" (UniqueName: \"kubernetes.io/projected/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-kube-api-access-gbnrv\") pod \"maas-api-revoke-sub-08ddf68f-rgfwh\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.551236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.551204 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ddf68f-rgfwh\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.551957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.551930 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ddf68f-rgfwh\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.553557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.553530 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ddf68f-rgfwh\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.581474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.581368 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbnrv\" (UniqueName: \"kubernetes.io/projected/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-kube-api-access-gbnrv\") pod \"maas-api-revoke-sub-08ddf68f-rgfwh\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.694174 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.694122 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:43.850374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:43.850218 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp"] Oct 30 01:19:44.156112 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.156012 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.241272 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.221454 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh"] Oct 30 01:19:44.404510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.404472 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" event={"ID":"01bd6e21-5e90-4063-956d-09f0b7523bd2","Type":"ContainerStarted","Data":"9baff3c621438c91f460a8e6bfdebb1d2ebfd060781e2d5ce1e0338b52c51b73"} Oct 30 01:19:44.404761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.404565 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" event={"ID":"01bd6e21-5e90-4063-956d-09f0b7523bd2","Type":"ContainerStarted","Data":"cbfc1e0211482a9ee6b790d6e9a8ece46d26d8c8a1c78d0ceac51c22997d7434"} Oct 30 01:19:44.404998 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.404972 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:44.406549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.406467 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" event={"ID":"1bb00a4d-eb99-4bc2-a152-ec99247ac45c","Type":"ContainerStarted","Data":"ae9f3dcce395dd6b6a1b86391fd771c249b7e400148426ad3eb75038d984d8ee"} Oct 30 01:19:44.406549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.406529 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" event={"ID":"1bb00a4d-eb99-4bc2-a152-ec99247ac45c","Type":"ContainerStarted","Data":"d1500de065adb9591f6c4e588bb9767a1fb52d449a235c8b186a993cf9e5e722"} Oct 30 01:19:44.406802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.406576 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:44.408818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.408791 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" event={"ID":"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc","Type":"ContainerStarted","Data":"dbaba843a8295c2dcfbf947a199a723986890dac0ab38e99ad3aa9588d0ba6b7"} Oct 30 01:19:44.448005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.447819 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" podStartSLOduration=2.447798538 podStartE2EDuration="2.447798538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:44.442486717 +0000 UTC m=+2722.354613856" watchObservedRunningTime="2026-10-30 01:19:44.447798538 +0000 UTC m=+2722.359925681" Oct 30 01:19:44.481437 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:44.481373 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" podStartSLOduration=2.481352936 podStartE2EDuration="2.481352936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:44.479611223 +0000 UTC m=+2722.391738365" watchObservedRunningTime="2026-10-30 01:19:44.481352936 +0000 UTC m=+2722.393480076" Oct 30 01:19:45.033818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.033685 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf"] Oct 30 01:19:45.039516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.039480 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e13d8d-68lrf"] Oct 30 01:19:45.121911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.121869 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:19:45.122085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.121946 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:19:45.364444 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.364401 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj"] Oct 30 01:19:45.364649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.364624 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" containerID="cri-o://05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834" gracePeriod=30 Oct 30 01:19:45.371171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.371127 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn"] Oct 30 01:19:45.371477 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.371448 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" containerID="cri-o://475347a3b6b814fb65cf25c9e9234b6db3a4ca532030284eda55487a2df87fae" gracePeriod=30 Oct 30 01:19:45.426363 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.426104 4517 generic.go:356] "Generic (PLEG): container finished" podID="a5fff1ac-4baa-44b2-8e01-f4d2eded83bc" containerID="19adbce1e28291954e14451fe842be4e19453868a8dbcd6fa646925c27291f9d" exitCode=0 Oct 30 01:19:45.426363 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.426155 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" event={"ID":"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc","Type":"ContainerDied","Data":"19adbce1e28291954e14451fe842be4e19453868a8dbcd6fa646925c27291f9d"} Oct 30 01:19:45.893837 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:45.893804 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:46.237977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.237894 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c06538-337b-41cc-ac02-e3eab004c9de" path="/var/lib/kubelet/pods/19c06538-337b-41cc-ac02-e3eab004c9de/volumes" Oct 30 01:19:46.678284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.678254 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:46.799731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.798086 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-openshift-ca\") pod \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " Oct 30 01:19:46.799731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.798240 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-xks-ca\") pod \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " Oct 30 01:19:46.799731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.798268 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-kube-api-access-gbnrv\" (UniqueName: \"kubernetes.io/projected/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-kube-api-access-gbnrv\") pod \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\" (UID: \"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc\") " Oct 30 01:19:46.799731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.799239 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-openshift-ca" pod "a5fff1ac-4baa-44b2-8e01-f4d2eded83bc" (UID: "a5fff1ac-4baa-44b2-8e01-f4d2eded83bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:46.808861 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.802253 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-kube-api-access-gbnrv" pod "a5fff1ac-4baa-44b2-8e01-f4d2eded83bc" (UID: "a5fff1ac-4baa-44b2-8e01-f4d2eded83bc"). InnerVolumeSpecName "kube-api-access-gbnrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:46.808861 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.802771 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-xks-ca" pod "a5fff1ac-4baa-44b2-8e01-f4d2eded83bc" (UID: "a5fff1ac-4baa-44b2-8e01-f4d2eded83bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:46.840973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.840940 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 01:19:46.900816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.900784 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:46.900816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.900813 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:46.901035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:46.900826 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbnrv\" (UniqueName: \"kubernetes.io/projected/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc-kube-api-access-gbnrv\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.003031 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-certs\") pod \"c5cb26e6-c654-445d-b430-28068567ffc3\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.003178 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-kube-api-access-ck444\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-kube-api-access-ck444\") pod \"c5cb26e6-c654-445d-b430-28068567ffc3\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.003315 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5cb26e6-c654-445d-b430-28068567ffc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb26e6-c654-445d-b430-28068567ffc3-istiod-ca-cert\") pod \"c5cb26e6-c654-445d-b430-28068567ffc3\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.003384 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-istio-token\") pod \"c5cb26e6-c654-445d-b430-28068567ffc3\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.003561 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-envoy\") pod \"c5cb26e6-c654-445d-b430-28068567ffc3\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.003613 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5cb26e6-c654-445d-b430-28068567ffc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb26e6-c654-445d-b430-28068567ffc3-istio-podinfo\") pod \"c5cb26e6-c654-445d-b430-28068567ffc3\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.003827 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-socket\") pod \"c5cb26e6-c654-445d-b430-28068567ffc3\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.003884 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-data\") pod \"c5cb26e6-c654-445d-b430-28068567ffc3\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.003953 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-credential-socket\") pod \"c5cb26e6-c654-445d-b430-28068567ffc3\" (UID: \"c5cb26e6-c654-445d-b430-28068567ffc3\") " Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.011883 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-credential-socket" pod "c5cb26e6-c654-445d-b430-28068567ffc3" (UID: "c5cb26e6-c654-445d-b430-28068567ffc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.011963 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-socket" pod "c5cb26e6-c654-445d-b430-28068567ffc3" (UID: "c5cb26e6-c654-445d-b430-28068567ffc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.012144 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-data" pod "c5cb26e6-c654-445d-b430-28068567ffc3" (UID: "c5cb26e6-c654-445d-b430-28068567ffc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.012314 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5cb26e6-c654-445d-b430-28068567ffc3-istiod-ca-cert" pod "c5cb26e6-c654-445d-b430-28068567ffc3" (UID: "c5cb26e6-c654-445d-b430-28068567ffc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:47.015672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.012596 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-certs" pod "c5cb26e6-c654-445d-b430-28068567ffc3" (UID: "c5cb26e6-c654-445d-b430-28068567ffc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:47.016510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.015962 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5cb26e6-c654-445d-b430-28068567ffc3-istio-podinfo" pod "c5cb26e6-c654-445d-b430-28068567ffc3" (UID: "c5cb26e6-c654-445d-b430-28068567ffc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:47.016510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.015964 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-istio-token" pod "c5cb26e6-c654-445d-b430-28068567ffc3" (UID: "c5cb26e6-c654-445d-b430-28068567ffc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:47.016510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.016066 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-kube-api-access-ck444" pod "c5cb26e6-c654-445d-b430-28068567ffc3" (UID: "c5cb26e6-c654-445d-b430-28068567ffc3"). InnerVolumeSpecName "kube-api-access-ck444". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:47.017719 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.017648 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-envoy" pod "c5cb26e6-c654-445d-b430-28068567ffc3" (UID: "c5cb26e6-c654-445d-b430-28068567ffc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:47.112751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.112644 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.112751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.112681 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck444\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-kube-api-access-ck444\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.112751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.112691 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cb26e6-c654-445d-b430-28068567ffc3-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.112751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.112701 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cb26e6-c654-445d-b430-28068567ffc3-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.112751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.112727 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.112751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.112735 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cb26e6-c654-445d-b430-28068567ffc3-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.112751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.112742 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.112751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.112752 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.112751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.112760 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cb26e6-c654-445d-b430-28068567ffc3-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:47.359751 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:47.357899 4517 logging.go:55] [core] [Channel #3759 SubChannel #3761]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.158:50052", ServerName: "10.133.0.158:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.158:50052: connect: connection reset by peer" Oct 30 01:19:47.359751 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:47.358042 4517 logging.go:55] [core] [Channel #3758 SubChannel #3760]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.157:50052", ServerName: "10.133.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.157:50052: connect: connection reset by peer" Oct 30 01:19:47.624735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.606580 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" event={"ID":"a5fff1ac-4baa-44b2-8e01-f4d2eded83bc","Type":"ContainerDied","Data":"dbaba843a8295c2dcfbf947a199a723986890dac0ab38e99ad3aa9588d0ba6b7"} Oct 30 01:19:47.624735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.606655 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbaba843a8295c2dcfbf947a199a723986890dac0ab38e99ad3aa9588d0ba6b7" Oct 30 01:19:47.721874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.649881 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh" Oct 30 01:19:47.721874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.674614 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:47.721874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.690754 4517 generic.go:356] "Generic (PLEG): container finished" podID="c5cb26e6-c654-445d-b430-28068567ffc3" containerID="1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038" exitCode=0 Oct 30 01:19:47.721874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.691114 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" Oct 30 01:19:47.721874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.704810 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" event={"ID":"c5cb26e6-c654-445d-b430-28068567ffc3","Type":"ContainerDied","Data":"1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038"} Oct 30 01:19:47.721874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.704897 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc" event={"ID":"c5cb26e6-c654-445d-b430-28068567ffc3","Type":"ContainerDied","Data":"47eb1d6a970b7390bbbde32bae7542118c0ac8a5194725ec69311f465d9087c8"} Oct 30 01:19:47.721874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.704935 4517 scope.go:122] "RemoveContainer" containerID="1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038" Oct 30 01:19:47.750595 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:47.750542 4517 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7ea040_73a1_4179_8b73_4ad5109192a8.slice/crio-7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7ea040_73a1_4179_8b73_4ad5109192a8.slice/crio-conmon-7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844c10a2_f543_4898_b69f_b207052e0d8d.slice/crio-31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cb26e6_c654_445d_b430_28068567ffc3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844c10a2_f543_4898_b69f_b207052e0d8d.slice/crio-conmon-31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:19:47.766525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.766199 4517 scope.go:122] "RemoveContainer" containerID="1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038" Oct 30 01:19:47.767296 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:47.767182 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038\": container with ID starting with 1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038 not found: ID does not exist" containerID="1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038" Oct 30 01:19:47.767296 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.767224 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038"} err="failed to get container status \"1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038\": rpc error: code = NotFound desc = could not find container \"1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038\": container with ID starting with 1ce888f44145ffc68abd569ae63c4563f694e54b5d04537a32fe3a1dfcbf3038 not found: ID does not exist" Oct 30 01:19:47.804432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.804367 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc"] Oct 30 01:19:47.816252 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:47.816115 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsffc"] Oct 30 01:19:48.017516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.017480 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" podUID="b3ad5174-8bc1-4097-a148-5a84af54c3e6" containerName="payload-processing" containerID="cri-o://90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f" gracePeriod=25 Oct 30 01:19:48.022167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.022136 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:19:48.039319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.039282 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm"] Oct 30 01:19:48.039588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.039541 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" containerID="cri-o://eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8" gracePeriod=30 Oct 30 01:19:48.124138 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.124107 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844c10a2-f543-4898-b69f-b207052e0d8d-kube-api-access-5z8hs\" (UniqueName: \"kubernetes.io/projected/844c10a2-f543-4898-b69f-b207052e0d8d-kube-api-access-5z8hs\") pod \"844c10a2-f543-4898-b69f-b207052e0d8d\" (UID: \"844c10a2-f543-4898-b69f-b207052e0d8d\") " Oct 30 01:19:48.124352 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.124190 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844c10a2-f543-4898-b69f-b207052e0d8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844c10a2-f543-4898-b69f-b207052e0d8d-plugins-config-volume\") pod \"844c10a2-f543-4898-b69f-b207052e0d8d\" (UID: \"844c10a2-f543-4898-b69f-b207052e0d8d\") " Oct 30 01:19:48.128227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.128189 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844c10a2-f543-4898-b69f-b207052e0d8d-kube-api-access-5z8hs" pod "844c10a2-f543-4898-b69f-b207052e0d8d" (UID: "844c10a2-f543-4898-b69f-b207052e0d8d"). InnerVolumeSpecName "kube-api-access-5z8hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:48.128865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.128830 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844c10a2-f543-4898-b69f-b207052e0d8d-plugins-config-volume" pod "844c10a2-f543-4898-b69f-b207052e0d8d" (UID: "844c10a2-f543-4898-b69f-b207052e0d8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:48.137556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.137489 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:19:48.196122 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:48.196085 4517 logging.go:55] [core] [Channel #3765 SubChannel #3766]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.204:50052", ServerName: "10.133.0.204:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.204:50052: connect: connection refused" Oct 30 01:19:48.242791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.242650 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7ea040-73a1-4179-8b73-4ad5109192a8-kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/3c7ea040-73a1-4179-8b73-4ad5109192a8-kube-api-access-rz2rx\") pod \"3c7ea040-73a1-4179-8b73-4ad5109192a8\" (UID: \"3c7ea040-73a1-4179-8b73-4ad5109192a8\") " Oct 30 01:19:48.242791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.242782 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7ea040-73a1-4179-8b73-4ad5109192a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7ea040-73a1-4179-8b73-4ad5109192a8-plugins-config-volume\") pod \"3c7ea040-73a1-4179-8b73-4ad5109192a8\" (UID: \"3c7ea040-73a1-4179-8b73-4ad5109192a8\") " Oct 30 01:19:48.244406 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.244371 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7ea040-73a1-4179-8b73-4ad5109192a8-plugins-config-volume" pod "3c7ea040-73a1-4179-8b73-4ad5109192a8" (UID: "3c7ea040-73a1-4179-8b73-4ad5109192a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:48.244641 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.244572 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z8hs\" (UniqueName: \"kubernetes.io/projected/844c10a2-f543-4898-b69f-b207052e0d8d-kube-api-access-5z8hs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:48.244641 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.244593 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844c10a2-f543-4898-b69f-b207052e0d8d-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:48.244641 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.244608 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7ea040-73a1-4179-8b73-4ad5109192a8-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:48.251768 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.251677 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7ea040-73a1-4179-8b73-4ad5109192a8-kube-api-access-rz2rx" pod "3c7ea040-73a1-4179-8b73-4ad5109192a8" (UID: "3c7ea040-73a1-4179-8b73-4ad5109192a8"). InnerVolumeSpecName "kube-api-access-rz2rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:48.259423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.259377 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" path="/var/lib/kubelet/pods/c5cb26e6-c654-445d-b430-28068567ffc3/volumes" Oct 30 01:19:48.334989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.334939 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" podUID="3c7ea040-73a1-4179-8b73-4ad5109192a8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.158:50052\" within 1s: context deadline exceeded" Oct 30 01:19:48.335222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.334939 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" podUID="844c10a2-f543-4898-b69f-b207052e0d8d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.157:50052\" within 1s: context deadline exceeded" Oct 30 01:19:48.346772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.346661 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/3c7ea040-73a1-4179-8b73-4ad5109192a8-kube-api-access-rz2rx\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:48.362432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.362305 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:19:48.448089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.448048 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ad5174-8bc1-4097-a148-5a84af54c3e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ad5174-8bc1-4097-a148-5a84af54c3e6-plugins-config-volume\") pod \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\" (UID: \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\") " Oct 30 01:19:48.448262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.448182 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ad5174-8bc1-4097-a148-5a84af54c3e6-kube-api-access-sbhxn\" (UniqueName: \"kubernetes.io/projected/b3ad5174-8bc1-4097-a148-5a84af54c3e6-kube-api-access-sbhxn\") pod \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\" (UID: \"b3ad5174-8bc1-4097-a148-5a84af54c3e6\") " Oct 30 01:19:48.448532 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.448508 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ad5174-8bc1-4097-a148-5a84af54c3e6-plugins-config-volume" pod "b3ad5174-8bc1-4097-a148-5a84af54c3e6" (UID: "b3ad5174-8bc1-4097-a148-5a84af54c3e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:48.448946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.448921 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ad5174-8bc1-4097-a148-5a84af54c3e6-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:48.450579 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.450551 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ad5174-8bc1-4097-a148-5a84af54c3e6-kube-api-access-sbhxn" pod "b3ad5174-8bc1-4097-a148-5a84af54c3e6" (UID: "b3ad5174-8bc1-4097-a148-5a84af54c3e6"). InnerVolumeSpecName "kube-api-access-sbhxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:48.550150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.550105 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhxn\" (UniqueName: \"kubernetes.io/projected/b3ad5174-8bc1-4097-a148-5a84af54c3e6-kube-api-access-sbhxn\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:48.703654 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.703611 4517 generic.go:356] "Generic (PLEG): container finished" podID="3c7ea040-73a1-4179-8b73-4ad5109192a8" containerID="7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5" exitCode=0 Oct 30 01:19:48.703889 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.703665 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" Oct 30 01:19:48.703889 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.703702 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" event={"ID":"3c7ea040-73a1-4179-8b73-4ad5109192a8","Type":"ContainerDied","Data":"7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5"} Oct 30 01:19:48.703889 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.703803 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj" event={"ID":"3c7ea040-73a1-4179-8b73-4ad5109192a8","Type":"ContainerDied","Data":"c75c0cd2006d969a5ffde2f4187d5d4b6ee31deda0f5b76a1100a980339d580c"} Oct 30 01:19:48.703889 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.703841 4517 scope.go:122] "RemoveContainer" containerID="7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5" Oct 30 01:19:48.705663 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.705590 4517 generic.go:356] "Generic (PLEG): container finished" podID="844c10a2-f543-4898-b69f-b207052e0d8d" containerID="31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50" exitCode=0 Oct 30 01:19:48.705663 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.705653 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" Oct 30 01:19:48.705888 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.705672 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" event={"ID":"844c10a2-f543-4898-b69f-b207052e0d8d","Type":"ContainerDied","Data":"31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50"} Oct 30 01:19:48.705888 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.705770 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf" event={"ID":"844c10a2-f543-4898-b69f-b207052e0d8d","Type":"ContainerDied","Data":"f0f41a8b94d2f8e5af70ad37fac17ddfcc52cbf0c39e79702592584619182090"} Oct 30 01:19:48.707917 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.707888 4517 generic.go:356] "Generic (PLEG): container finished" podID="b3ad5174-8bc1-4097-a148-5a84af54c3e6" containerID="90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f" exitCode=0 Oct 30 01:19:48.708037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.707929 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" event={"ID":"b3ad5174-8bc1-4097-a148-5a84af54c3e6","Type":"ContainerDied","Data":"90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f"} Oct 30 01:19:48.708037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.707988 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" event={"ID":"b3ad5174-8bc1-4097-a148-5a84af54c3e6","Type":"ContainerDied","Data":"21f4d8e385881bf5126ebb4fc52926efaad35e2d0ea344078d450a030eb98bd0"} Oct 30 01:19:48.708037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.707937 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" Oct 30 01:19:48.716271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.716229 4517 scope.go:122] "RemoveContainer" containerID="7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5" Oct 30 01:19:48.717105 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:48.717081 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5\": container with ID starting with 7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5 not found: ID does not exist" containerID="7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5" Oct 30 01:19:48.717202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.717113 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5"} err="failed to get container status \"7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5\": rpc error: code = NotFound desc = could not find container \"7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5\": container with ID starting with 7f1b1b5fb043cb89689c008450b02faf2290799bd7a816ba0b4963716b6a72a5 not found: ID does not exist" Oct 30 01:19:48.717202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.717132 4517 scope.go:122] "RemoveContainer" containerID="31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50" Oct 30 01:19:48.728669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.728647 4517 scope.go:122] "RemoveContainer" containerID="31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50" Oct 30 01:19:48.729156 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:48.729135 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50\": container with ID starting with 31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50 not found: ID does not exist" containerID="31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50" Oct 30 01:19:48.729237 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.729170 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50"} err="failed to get container status \"31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50\": rpc error: code = NotFound desc = could not find container \"31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50\": container with ID starting with 31f3af3e58f35df52fd74ffee2461887aea133eed245d04ac40d3c5113a92b50 not found: ID does not exist" Oct 30 01:19:48.729237 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.729196 4517 scope.go:122] "RemoveContainer" containerID="90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f" Oct 30 01:19:48.741097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.740959 4517 scope.go:122] "RemoveContainer" containerID="90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f" Oct 30 01:19:48.741352 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:48.741331 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f\": container with ID starting with 90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f not found: ID does not exist" containerID="90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f" Oct 30 01:19:48.741417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.741367 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f"} err="failed to get container status \"90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f\": rpc error: code = NotFound desc = could not find container \"90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f\": container with ID starting with 90e0f9a6f66262121c10366d8b944880ee4e92d76ec1bd5950657343e7ff390f not found: ID does not exist" Oct 30 01:19:48.757411 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.757371 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c"] Oct 30 01:19:48.765748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.765679 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c"] Oct 30 01:19:48.809400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.809297 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf"] Oct 30 01:19:48.815034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.814987 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f69104-698c567f7f-dmphf"] Oct 30 01:19:48.845327 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.845278 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj"] Oct 30 01:19:48.850279 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:48.850238 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f69104-7cf4dc59cb-cb6sj"] Oct 30 01:19:49.195695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:49.195573 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-5d787c7c54-v5z7c" podUID="b3ad5174-8bc1-4097-a148-5a84af54c3e6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.204:50052\" within 1s: context deadline exceeded" Oct 30 01:19:49.492256 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:49.492152 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:50.237265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.237221 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7ea040-73a1-4179-8b73-4ad5109192a8" path="/var/lib/kubelet/pods/3c7ea040-73a1-4179-8b73-4ad5109192a8/volumes" Oct 30 01:19:50.237883 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.237859 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844c10a2-f543-4898-b69f-b207052e0d8d" path="/var/lib/kubelet/pods/844c10a2-f543-4898-b69f-b207052e0d8d/volumes" Oct 30 01:19:50.238416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.238398 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ad5174-8bc1-4097-a148-5a84af54c3e6" path="/var/lib/kubelet/pods/b3ad5174-8bc1-4097-a148-5a84af54c3e6/volumes" Oct 30 01:19:50.709925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.709888 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:19:50.733048 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.733009 4517 generic.go:356] "Generic (PLEG): container finished" podID="07671c93-8322-4ca8-a227-10163344a3b1" containerID="05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834" exitCode=0 Oct 30 01:19:50.733248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.733064 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" Oct 30 01:19:50.733248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.733091 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" event={"ID":"07671c93-8322-4ca8-a227-10163344a3b1","Type":"ContainerDied","Data":"05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834"} Oct 30 01:19:50.733248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.733163 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj" event={"ID":"07671c93-8322-4ca8-a227-10163344a3b1","Type":"ContainerDied","Data":"bdebc1035ecbf4efab62666940b181cf95088091dde1d6e6ebe06cf845c80e8a"} Oct 30 01:19:50.733248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.733188 4517 scope.go:122] "RemoveContainer" containerID="05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834" Oct 30 01:19:50.735249 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.735215 4517 generic.go:356] "Generic (PLEG): container finished" podID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerID="475347a3b6b814fb65cf25c9e9234b6db3a4ca532030284eda55487a2df87fae" exitCode=0 Oct 30 01:19:50.735404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.735364 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" event={"ID":"d1f99f06-6df7-4e18-9154-e37dee7d6f70","Type":"ContainerDied","Data":"475347a3b6b814fb65cf25c9e9234b6db3a4ca532030284eda55487a2df87fae"} Oct 30 01:19:50.755443 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.755365 4517 scope.go:122] "RemoveContainer" containerID="05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834" Oct 30 01:19:50.756526 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:50.755854 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834\": container with ID starting with 05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834 not found: ID does not exist" containerID="05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834" Oct 30 01:19:50.756526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.755898 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834"} err="failed to get container status \"05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834\": rpc error: code = NotFound desc = could not find container \"05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834\": container with ID starting with 05ee95f11ffe65ad414fdaf91f7ae1d514d7a718816d56b5ad1b7d2837d6d834 not found: ID does not exist" Oct 30 01:19:50.759420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.759384 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:19:50.793990 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.793945 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-istio-token\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-istio-token\") pod \"07671c93-8322-4ca8-a227-10163344a3b1\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " Oct 30 01:19:50.794192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.794025 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-credential-socket\") pod \"07671c93-8322-4ca8-a227-10163344a3b1\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " Oct 30 01:19:50.794192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.794086 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-data\") pod \"07671c93-8322-4ca8-a227-10163344a3b1\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " Oct 30 01:19:50.794192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.794118 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-socket\") pod \"07671c93-8322-4ca8-a227-10163344a3b1\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " Oct 30 01:19:50.794192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.794161 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-certs\") pod \"07671c93-8322-4ca8-a227-10163344a3b1\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " Oct 30 01:19:50.794469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.794315 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07671c93-8322-4ca8-a227-10163344a3b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07671c93-8322-4ca8-a227-10163344a3b1-istiod-ca-cert\") pod \"07671c93-8322-4ca8-a227-10163344a3b1\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " Oct 30 01:19:50.794469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.794428 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07671c93-8322-4ca8-a227-10163344a3b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07671c93-8322-4ca8-a227-10163344a3b1-istio-podinfo\") pod \"07671c93-8322-4ca8-a227-10163344a3b1\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " Oct 30 01:19:50.794582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.794473 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-envoy\") pod \"07671c93-8322-4ca8-a227-10163344a3b1\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " Oct 30 01:19:50.794645 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.794568 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-kube-api-access-gzjx6\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-kube-api-access-gzjx6\") pod \"07671c93-8322-4ca8-a227-10163344a3b1\" (UID: \"07671c93-8322-4ca8-a227-10163344a3b1\") " Oct 30 01:19:50.796391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.796355 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-data" pod "07671c93-8322-4ca8-a227-10163344a3b1" (UID: "07671c93-8322-4ca8-a227-10163344a3b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.797489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.797455 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-certs" pod "07671c93-8322-4ca8-a227-10163344a3b1" (UID: "07671c93-8322-4ca8-a227-10163344a3b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.798054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.798030 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-credential-socket" pod "07671c93-8322-4ca8-a227-10163344a3b1" (UID: "07671c93-8322-4ca8-a227-10163344a3b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.798166 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.798064 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07671c93-8322-4ca8-a227-10163344a3b1-istiod-ca-cert" pod "07671c93-8322-4ca8-a227-10163344a3b1" (UID: "07671c93-8322-4ca8-a227-10163344a3b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:50.798262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.798243 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-socket" pod "07671c93-8322-4ca8-a227-10163344a3b1" (UID: "07671c93-8322-4ca8-a227-10163344a3b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.799454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.799413 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-kube-api-access-gzjx6" pod "07671c93-8322-4ca8-a227-10163344a3b1" (UID: "07671c93-8322-4ca8-a227-10163344a3b1"). InnerVolumeSpecName "kube-api-access-gzjx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:50.807232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.807180 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-istio-token" pod "07671c93-8322-4ca8-a227-10163344a3b1" (UID: "07671c93-8322-4ca8-a227-10163344a3b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:50.808395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.808355 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07671c93-8322-4ca8-a227-10163344a3b1-istio-podinfo" pod "07671c93-8322-4ca8-a227-10163344a3b1" (UID: "07671c93-8322-4ca8-a227-10163344a3b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:50.809171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.809142 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-envoy" pod "07671c93-8322-4ca8-a227-10163344a3b1" (UID: "07671c93-8322-4ca8-a227-10163344a3b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.896530 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.896490 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-data\") pod \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " Oct 30 01:19:50.896530 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.896535 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istiod-ca-cert\") pod \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " Oct 30 01:19:50.896879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.896650 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-credential-socket\") pod \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " Oct 30 01:19:50.896879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.896771 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-certs\") pod \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " Oct 30 01:19:50.896879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.896812 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-token\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-token\") pod \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " Oct 30 01:19:50.896879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.896852 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-podinfo\") pod \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " Oct 30 01:19:50.897125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.896925 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-data" pod "d1f99f06-6df7-4e18-9154-e37dee7d6f70" (UID: "d1f99f06-6df7-4e18-9154-e37dee7d6f70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.897125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.896965 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-socket\") pod \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " Oct 30 01:19:50.897125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.897000 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-envoy\") pod \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " Oct 30 01:19:50.898109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.897129 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-kube-api-access-s79cg\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-kube-api-access-s79cg\") pod \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\" (UID: \"d1f99f06-6df7-4e18-9154-e37dee7d6f70\") " Oct 30 01:19:50.898109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.897465 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-certs" pod "d1f99f06-6df7-4e18-9154-e37dee7d6f70" (UID: "d1f99f06-6df7-4e18-9154-e37dee7d6f70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.898109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.897377 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istiod-ca-cert" pod "d1f99f06-6df7-4e18-9154-e37dee7d6f70" (UID: "d1f99f06-6df7-4e18-9154-e37dee7d6f70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:50.898109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.897492 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-credential-socket" pod "d1f99f06-6df7-4e18-9154-e37dee7d6f70" (UID: "d1f99f06-6df7-4e18-9154-e37dee7d6f70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.898109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.897887 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-socket" pod "d1f99f06-6df7-4e18-9154-e37dee7d6f70" (UID: "d1f99f06-6df7-4e18-9154-e37dee7d6f70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.898439 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898365 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898439 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898388 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898448 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898464 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898477 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898490 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07671c93-8322-4ca8-a227-10163344a3b1-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898505 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898518 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898532 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07671c93-8322-4ca8-a227-10163344a3b1-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898544 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.898556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898556 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.899133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898572 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzjx6\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-kube-api-access-gzjx6\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.899133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898588 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07671c93-8322-4ca8-a227-10163344a3b1-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.899133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.898601 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07671c93-8322-4ca8-a227-10163344a3b1-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.899945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.899910 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-podinfo" pod "d1f99f06-6df7-4e18-9154-e37dee7d6f70" (UID: "d1f99f06-6df7-4e18-9154-e37dee7d6f70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:50.900063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.900023 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-token" pod "d1f99f06-6df7-4e18-9154-e37dee7d6f70" (UID: "d1f99f06-6df7-4e18-9154-e37dee7d6f70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:50.900333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.900308 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-envoy" pod "d1f99f06-6df7-4e18-9154-e37dee7d6f70" (UID: "d1f99f06-6df7-4e18-9154-e37dee7d6f70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:50.900390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:50.900337 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-kube-api-access-s79cg" pod "d1f99f06-6df7-4e18-9154-e37dee7d6f70" (UID: "d1f99f06-6df7-4e18-9154-e37dee7d6f70"). InnerVolumeSpecName "kube-api-access-s79cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:51.000549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.000503 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s79cg\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-kube-api-access-s79cg\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:51.000549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.000538 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:51.000549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.000550 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:51.000871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.000562 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f99f06-6df7-4e18-9154-e37dee7d6f70-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:51.078934 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.078892 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj"] Oct 30 01:19:51.089162 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.089127 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-x2mfj"] Oct 30 01:19:51.214218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.214174 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:51.745985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.745891 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" Oct 30 01:19:51.745985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.745908 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn" event={"ID":"d1f99f06-6df7-4e18-9154-e37dee7d6f70","Type":"ContainerDied","Data":"09017c1d6a686424c9a371d5466119af7d25588f97b1c3977787c3f4810f0d1d"} Oct 30 01:19:51.745985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.745979 4517 scope.go:122] "RemoveContainer" containerID="475347a3b6b814fb65cf25c9e9234b6db3a4ca532030284eda55487a2df87fae" Oct 30 01:19:51.795791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.795753 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn"] Oct 30 01:19:51.808405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:51.808359 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-lsmcn"] Oct 30 01:19:52.237975 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:52.237941 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07671c93-8322-4ca8-a227-10163344a3b1" path="/var/lib/kubelet/pods/07671c93-8322-4ca8-a227-10163344a3b1/volumes" Oct 30 01:19:52.238525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:52.238510 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" path="/var/lib/kubelet/pods/d1f99f06-6df7-4e18-9154-e37dee7d6f70/volumes" Oct 30 01:19:52.847523 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:52.847486 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:53.354587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.354552 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:19:53.389953 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.389921 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:19:53.391509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.391485 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:19:53.441444 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.441344 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-envoy\") pod \"04fe6ff0-38aa-47bd-977d-12969d6ead64\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " Oct 30 01:19:53.441444 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.441414 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-kube-api-access-d85cw\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-kube-api-access-d85cw\") pod \"04fe6ff0-38aa-47bd-977d-12969d6ead64\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " Oct 30 01:19:53.442376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.441480 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-certs\") pod \"04fe6ff0-38aa-47bd-977d-12969d6ead64\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " Oct 30 01:19:53.442376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.441512 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-credential-socket\") pod \"04fe6ff0-38aa-47bd-977d-12969d6ead64\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " Oct 30 01:19:53.442376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.441618 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fe6ff0-38aa-47bd-977d-12969d6ead64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe6ff0-38aa-47bd-977d-12969d6ead64-istiod-ca-cert\") pod \"04fe6ff0-38aa-47bd-977d-12969d6ead64\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " Oct 30 01:19:53.442376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.441651 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-data\") pod \"04fe6ff0-38aa-47bd-977d-12969d6ead64\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " Oct 30 01:19:53.442376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.442115 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-socket\") pod \"04fe6ff0-38aa-47bd-977d-12969d6ead64\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " Oct 30 01:19:53.442376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.442226 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-credential-socket" pod "04fe6ff0-38aa-47bd-977d-12969d6ead64" (UID: "04fe6ff0-38aa-47bd-977d-12969d6ead64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.442376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.442282 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-token\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-token\") pod \"04fe6ff0-38aa-47bd-977d-12969d6ead64\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " Oct 30 01:19:53.442376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.442325 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-podinfo\") pod \"04fe6ff0-38aa-47bd-977d-12969d6ead64\" (UID: \"04fe6ff0-38aa-47bd-977d-12969d6ead64\") " Oct 30 01:19:53.442376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.442330 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fe6ff0-38aa-47bd-977d-12969d6ead64-istiod-ca-cert" pod "04fe6ff0-38aa-47bd-977d-12969d6ead64" (UID: "04fe6ff0-38aa-47bd-977d-12969d6ead64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:53.444776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.442423 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-data" pod "04fe6ff0-38aa-47bd-977d-12969d6ead64" (UID: "04fe6ff0-38aa-47bd-977d-12969d6ead64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.444776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.442571 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-certs" pod "04fe6ff0-38aa-47bd-977d-12969d6ead64" (UID: "04fe6ff0-38aa-47bd-977d-12969d6ead64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.444776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.442684 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-socket" pod "04fe6ff0-38aa-47bd-977d-12969d6ead64" (UID: "04fe6ff0-38aa-47bd-977d-12969d6ead64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.445124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.444982 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.445124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.445008 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.445124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.445024 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fe6ff0-38aa-47bd-977d-12969d6ead64-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.445124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.445037 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.445124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.445052 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.445585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.445497 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-envoy" pod "04fe6ff0-38aa-47bd-977d-12969d6ead64" (UID: "04fe6ff0-38aa-47bd-977d-12969d6ead64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.447576 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.447541 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-podinfo" pod "04fe6ff0-38aa-47bd-977d-12969d6ead64" (UID: "04fe6ff0-38aa-47bd-977d-12969d6ead64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:53.447748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.447545 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-kube-api-access-d85cw" pod "04fe6ff0-38aa-47bd-977d-12969d6ead64" (UID: "04fe6ff0-38aa-47bd-977d-12969d6ead64"). InnerVolumeSpecName "kube-api-access-d85cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:53.448862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.448839 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-token" pod "04fe6ff0-38aa-47bd-977d-12969d6ead64" (UID: "04fe6ff0-38aa-47bd-977d-12969d6ead64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:53.547152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.547111 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.547152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.547145 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.547152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.547158 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fe6ff0-38aa-47bd-977d-12969d6ead64-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.547495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.547166 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d85cw\" (UniqueName: \"kubernetes.io/projected/04fe6ff0-38aa-47bd-977d-12969d6ead64-kube-api-access-d85cw\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.769604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.769496 4517 generic.go:356] "Generic (PLEG): container finished" podID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerID="eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8" exitCode=0 Oct 30 01:19:53.769604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.769566 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" Oct 30 01:19:53.769833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.769594 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" event={"ID":"04fe6ff0-38aa-47bd-977d-12969d6ead64","Type":"ContainerDied","Data":"eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8"} Oct 30 01:19:53.769833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.769667 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm" event={"ID":"04fe6ff0-38aa-47bd-977d-12969d6ead64","Type":"ContainerDied","Data":"ec56a28e7618b52b9c1135f128f677d75e02d9406c2d16e63e0e9f0004e83b37"} Oct 30 01:19:53.769833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.769698 4517 scope.go:122] "RemoveContainer" containerID="eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8" Oct 30 01:19:53.785392 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.785205 4517 scope.go:122] "RemoveContainer" containerID="eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8" Oct 30 01:19:53.785669 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:19:53.785641 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8\": container with ID starting with eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8 not found: ID does not exist" containerID="eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8" Oct 30 01:19:53.786818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.785684 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8"} err="failed to get container status \"eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8\": rpc error: code = NotFound desc = could not find container \"eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8\": container with ID starting with eadfaaf2261fcd35b8b449821c2ba51797e22837118d4f2ad86fd8add1cd86f8 not found: ID does not exist" Oct 30 01:19:53.818946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.818870 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm"] Oct 30 01:19:53.830029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:53.829983 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-qxdsm"] Oct 30 01:19:54.218999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:54.218935 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:54.238202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:54.238158 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" path="/var/lib/kubelet/pods/04fe6ff0-38aa-47bd-977d-12969d6ead64/volumes" Oct 30 01:19:54.687782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:54.687750 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:55.017868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.017749 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:55.216487 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.216442 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f"] Oct 30 01:19:55.217832 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.217796 4517 cpu_manager.go:414] "containerMap: removing container" podUID="844c10a2-f543-4898-b69f-b207052e0d8d" containerName="payload-pre-processing" Oct 30 01:19:55.218075 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.218061 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844c10a2-f543-4898-b69f-b207052e0d8d" containerName="payload-pre-processing" Oct 30 01:19:55.218178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.218169 4517 cpu_manager.go:414] "containerMap: removing container" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" Oct 30 01:19:55.218242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.218233 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" Oct 30 01:19:55.218309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.218300 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a5fff1ac-4baa-44b2-8e01-f4d2eded83bc" containerName="revoke-keys" Oct 30 01:19:55.218378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.218369 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fff1ac-4baa-44b2-8e01-f4d2eded83bc" containerName="revoke-keys" Oct 30 01:19:55.218465 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.218457 4517 cpu_manager.go:414] "containerMap: removing container" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" Oct 30 01:19:55.218532 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.218523 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" Oct 30 01:19:55.218607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.218599 4517 cpu_manager.go:414] "containerMap: removing container" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" Oct 30 01:19:55.218665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.218657 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" Oct 30 01:19:55.224871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.224829 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b3ad5174-8bc1-4097-a148-5a84af54c3e6" containerName="payload-processing" Oct 30 01:19:55.225469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.225445 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ad5174-8bc1-4097-a148-5a84af54c3e6" containerName="payload-processing" Oct 30 01:19:55.225582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.225570 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" Oct 30 01:19:55.225665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.225654 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" Oct 30 01:19:55.225809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.225797 4517 cpu_manager.go:414] "containerMap: removing container" podUID="3c7ea040-73a1-4179-8b73-4ad5109192a8" containerName="payload-processing" Oct 30 01:19:55.225905 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.225893 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7ea040-73a1-4179-8b73-4ad5109192a8" containerName="payload-processing" Oct 30 01:19:55.226307 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.226287 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fe6ff0-38aa-47bd-977d-12969d6ead64" containerName="istio-proxy" Oct 30 01:19:55.227939 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.227915 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cb26e6-c654-445d-b430-28068567ffc3" containerName="istio-proxy" Oct 30 01:19:55.228187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.228168 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ad5174-8bc1-4097-a148-5a84af54c3e6" containerName="payload-processing" Oct 30 01:19:55.228323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.228312 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f99f06-6df7-4e18-9154-e37dee7d6f70" containerName="istio-proxy" Oct 30 01:19:55.228444 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.228433 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7ea040-73a1-4179-8b73-4ad5109192a8" containerName="payload-processing" Oct 30 01:19:55.228557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.228546 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="844c10a2-f543-4898-b69f-b207052e0d8d" containerName="payload-pre-processing" Oct 30 01:19:55.228656 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.228646 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="07671c93-8322-4ca8-a227-10163344a3b1" containerName="istio-proxy" Oct 30 01:19:55.228788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.228776 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fff1ac-4baa-44b2-8e01-f4d2eded83bc" containerName="revoke-keys" Oct 30 01:19:55.241120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.236303 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.245985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.245942 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f"] Oct 30 01:19:55.269028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.268931 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:19:55.269028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.268988 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:19:55.417451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.417403 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.417663 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.417624 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.418112 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.418082 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdjml\" (UniqueName: \"kubernetes.io/projected/717e8a03-586b-41c3-8630-bc67ff068221-kube-api-access-cdjml\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.430582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.430550 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:19:55.431533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.431504 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:19:55.520103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.520003 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.520103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.520101 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.520353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.520307 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdjml\" (UniqueName: \"kubernetes.io/projected/717e8a03-586b-41c3-8630-bc67ff068221-kube-api-access-cdjml\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.521689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.521654 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.529095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.529048 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.544406 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.544378 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdjml\" (UniqueName: \"kubernetes.io/projected/717e8a03-586b-41c3-8630-bc67ff068221-kube-api-access-cdjml\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.578846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.578809 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:55.901844 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:55.901809 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f"] Oct 30 01:19:55.903019 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:19:55.902985 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717e8a03_586b_41c3_8630_bc67ff068221.slice/crio-44d52777422fd45211050270a10b497e539d0a8d01b2556802e3ac6f5a15fcd9 WatchSource:0}: Error finding container 44d52777422fd45211050270a10b497e539d0a8d01b2556802e3ac6f5a15fcd9: Status 404 returned error can't find the container with id 44d52777422fd45211050270a10b497e539d0a8d01b2556802e3ac6f5a15fcd9 Oct 30 01:19:56.550979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:56.550935 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:56.799436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:56.799390 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" event={"ID":"717e8a03-586b-41c3-8630-bc67ff068221","Type":"ContainerStarted","Data":"6955bc015ead17d18d321d727d66850312220da922630ba43a0be15ac003c5b7"} Oct 30 01:19:56.799630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:56.799463 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" event={"ID":"717e8a03-586b-41c3-8630-bc67ff068221","Type":"ContainerStarted","Data":"44d52777422fd45211050270a10b497e539d0a8d01b2556802e3ac6f5a15fcd9"} Oct 30 01:19:56.839792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:56.839665 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" podStartSLOduration=1.8396469880000001 podStartE2EDuration="1.839646988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:56.835315074 +0000 UTC m=+2734.747442214" watchObservedRunningTime="2026-10-30 01:19:56.839646988 +0000 UTC m=+2734.751774128" Oct 30 01:19:57.808946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:57.808904 4517 generic.go:356] "Generic (PLEG): container finished" podID="717e8a03-586b-41c3-8630-bc67ff068221" containerID="6955bc015ead17d18d321d727d66850312220da922630ba43a0be15ac003c5b7" exitCode=0 Oct 30 01:19:57.809382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:57.808983 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" event={"ID":"717e8a03-586b-41c3-8630-bc67ff068221","Type":"ContainerDied","Data":"6955bc015ead17d18d321d727d66850312220da922630ba43a0be15ac003c5b7"} Oct 30 01:19:59.005487 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.005457 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:59.076695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.076598 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:59.086535 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.086482 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-openshift-ca\") pod \"717e8a03-586b-41c3-8630-bc67ff068221\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " Oct 30 01:19:59.087008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.086629 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717e8a03-586b-41c3-8630-bc67ff068221-kube-api-access-cdjml\" (UniqueName: \"kubernetes.io/projected/717e8a03-586b-41c3-8630-bc67ff068221-kube-api-access-cdjml\") pod \"717e8a03-586b-41c3-8630-bc67ff068221\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " Oct 30 01:19:59.087008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.086674 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-xks-ca\") pod \"717e8a03-586b-41c3-8630-bc67ff068221\" (UID: \"717e8a03-586b-41c3-8630-bc67ff068221\") " Oct 30 01:19:59.087197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.087093 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-openshift-ca" pod "717e8a03-586b-41c3-8630-bc67ff068221" (UID: "717e8a03-586b-41c3-8630-bc67ff068221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:59.089784 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.088255 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:59.098726 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.098655 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717e8a03-586b-41c3-8630-bc67ff068221-kube-api-access-cdjml" pod "717e8a03-586b-41c3-8630-bc67ff068221" (UID: "717e8a03-586b-41c3-8630-bc67ff068221"). InnerVolumeSpecName "kube-api-access-cdjml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:59.101788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.099930 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-xks-ca" pod "717e8a03-586b-41c3-8630-bc67ff068221" (UID: "717e8a03-586b-41c3-8630-bc67ff068221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:59.190517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.190478 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdjml\" (UniqueName: \"kubernetes.io/projected/717e8a03-586b-41c3-8630-bc67ff068221-kube-api-access-cdjml\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:59.190517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.190516 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717e8a03-586b-41c3-8630-bc67ff068221-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:19:59.828558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.828526 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" Oct 30 01:19:59.828784 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.828558 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f" event={"ID":"717e8a03-586b-41c3-8630-bc67ff068221","Type":"ContainerDied","Data":"44d52777422fd45211050270a10b497e539d0a8d01b2556802e3ac6f5a15fcd9"} Oct 30 01:19:59.828784 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:19:59.828619 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44d52777422fd45211050270a10b497e539d0a8d01b2556802e3ac6f5a15fcd9" Oct 30 01:20:00.762176 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:20:00.762132 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:20:00.882109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:00.882073 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:02.528639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:02.528553 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:03.316223 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:03.316181 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:04.161631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:04.161588 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:05.806324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:05.806285 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:05.925738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:05.925518 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:06.393731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:06.379624 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:08.067132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:08.067083 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:09.929512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:09.929468 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:10.438735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:10.434502 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:10.665454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:10.665397 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:10.683097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:10.683050 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb"] Oct 30 01:20:10.690926 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:10.690818 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f69104-k9cmb"] Oct 30 01:20:10.818767 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:10.818729 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:10.939335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:10.939302 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:11.167801 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:11.167764 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:12.238669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:12.238619 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722ffa59-fa18-488c-9a39-5a4f0d38b5c2" path="/var/lib/kubelet/pods/722ffa59-fa18-488c-9a39-5a4f0d38b5c2/volumes" Oct 30 01:20:12.613544 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:12.613503 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:14.131840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:14.131794 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:14.328454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:14.328355 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:14.465265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:14.465162 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:14.792807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:14.773401 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:14.935772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:14.935215 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:15.032282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:15.032236 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x"] Oct 30 01:20:15.039611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:15.039575 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f91e87-jsv4x"] Oct 30 01:20:15.074043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:15.073858 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:15.456627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:15.456530 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:16.237679 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:16.237636 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07ce999-326b-4fcd-998b-abf089f12248" path="/var/lib/kubelet/pods/f07ce999-326b-4fcd-998b-abf089f12248/volumes" Oct 30 01:20:17.006441 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:17.006404 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:17.650886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:17.650850 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:17.766254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:17.766215 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:18.076819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.059507 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7"] Oct 30 01:20:18.126779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.104338 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:18.151775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.149445 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" containerID="cri-o://d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5" gracePeriod=30 Oct 30 01:20:18.615445 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.577784 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:20:18.630621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.630582 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:18.778733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.777998 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:18.796131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.796051 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85"] Oct 30 01:20:18.797518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.797488 4517 cpu_manager.go:414] "containerMap: removing container" podUID="717e8a03-586b-41c3-8630-bc67ff068221" containerName="revoke-keys" Oct 30 01:20:18.797698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.797685 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717e8a03-586b-41c3-8630-bc67ff068221" containerName="revoke-keys" Oct 30 01:20:18.798159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.798138 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="717e8a03-586b-41c3-8630-bc67ff068221" containerName="revoke-keys" Oct 30 01:20:18.806342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.806294 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:18.824833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.824799 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85"] Oct 30 01:20:18.845624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.845586 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:20:18.845912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.845641 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:20:18.987502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.987401 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:18.987674 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.987628 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:18.987788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:18.987773 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtsn\" (UniqueName: \"kubernetes.io/projected/115ac86b-e232-4ac4-93ce-72e8a0d20541-kube-api-access-frtsn\") pod \"maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:19.089695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:19.089657 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:19.090264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:19.089791 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtsn\" (UniqueName: \"kubernetes.io/projected/115ac86b-e232-4ac4-93ce-72e8a0d20541-kube-api-access-frtsn\") pod \"maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:19.090264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:19.089945 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:19.090443 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:19.090420 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:19.092789 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:19.092763 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:19.112589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:19.112519 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtsn\" (UniqueName: \"kubernetes.io/projected/115ac86b-e232-4ac4-93ce-72e8a0d20541-kube-api-access-frtsn\") pod \"maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:19.133248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:19.132882 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:19.450517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:19.450484 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:19.453660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:19.453627 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85"] Oct 30 01:20:19.455227 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:20:19.455199 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115ac86b_e232_4ac4_93ce_72e8a0d20541.slice/crio-1e464e63c55441bac1622a0dbe0391e260d92883904fc11377e9148c34a77a1e WatchSource:0}: Error finding container 1e464e63c55441bac1622a0dbe0391e260d92883904fc11377e9148c34a77a1e: Status 404 returned error can't find the container with id 1e464e63c55441bac1622a0dbe0391e260d92883904fc11377e9148c34a77a1e Oct 30 01:20:20.179841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:20.179751 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" event={"ID":"115ac86b-e232-4ac4-93ce-72e8a0d20541","Type":"ContainerStarted","Data":"ec0a148f978dacd753b8314eb718a6ad3a5f63b773ea73cab1a45be711aa5f60"} Oct 30 01:20:20.179841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:20.179845 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" event={"ID":"115ac86b-e232-4ac4-93ce-72e8a0d20541","Type":"ContainerStarted","Data":"1e464e63c55441bac1622a0dbe0391e260d92883904fc11377e9148c34a77a1e"} Oct 30 01:20:20.218220 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:20.218160 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" podStartSLOduration=2.218139394 podStartE2EDuration="2.218139394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:20.216451512 +0000 UTC m=+2758.128578651" watchObservedRunningTime="2026-10-30 01:20:20.218139394 +0000 UTC m=+2758.130266535" Oct 30 01:20:21.192309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:21.192262 4517 generic.go:356] "Generic (PLEG): container finished" podID="115ac86b-e232-4ac4-93ce-72e8a0d20541" containerID="ec0a148f978dacd753b8314eb718a6ad3a5f63b773ea73cab1a45be711aa5f60" exitCode=0 Oct 30 01:20:21.193006 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:21.192401 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" event={"ID":"115ac86b-e232-4ac4-93ce-72e8a0d20541","Type":"ContainerDied","Data":"ec0a148f978dacd753b8314eb718a6ad3a5f63b773ea73cab1a45be711aa5f60"} Oct 30 01:20:21.328052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:21.328008 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:21.577963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:21.577927 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:21.693189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:21.693154 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:22.059922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.059885 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:22.439502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.439317 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk"] Oct 30 01:20:22.456479 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.456427 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.469305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.467486 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk"] Oct 30 01:20:22.509786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.509654 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvp68\" (UniqueName: \"kubernetes.io/projected/189348f5-35d5-4a46-a3ee-c28513b27176-kube-api-access-qvp68\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.510370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.510252 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.510690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.510533 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.612586 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.612537 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvp68\" (UniqueName: \"kubernetes.io/projected/189348f5-35d5-4a46-a3ee-c28513b27176-kube-api-access-qvp68\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.612866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.612847 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.612982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.612956 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.613899 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.613797 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.615953 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.615919 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.634942 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.634850 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvp68\" (UniqueName: \"kubernetes.io/projected/189348f5-35d5-4a46-a3ee-c28513b27176-kube-api-access-qvp68\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.678453 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.678416 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:22.816560 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.816491 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-xks-ca\") pod \"115ac86b-e232-4ac4-93ce-72e8a0d20541\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " Oct 30 01:20:22.816848 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.816592 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-openshift-ca\") pod \"115ac86b-e232-4ac4-93ce-72e8a0d20541\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " Oct 30 01:20:22.816936 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.816901 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115ac86b-e232-4ac4-93ce-72e8a0d20541-kube-api-access-frtsn\" (UniqueName: \"kubernetes.io/projected/115ac86b-e232-4ac4-93ce-72e8a0d20541-kube-api-access-frtsn\") pod \"115ac86b-e232-4ac4-93ce-72e8a0d20541\" (UID: \"115ac86b-e232-4ac4-93ce-72e8a0d20541\") " Oct 30 01:20:22.817033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.817002 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-openshift-ca" pod "115ac86b-e232-4ac4-93ce-72e8a0d20541" (UID: "115ac86b-e232-4ac4-93ce-72e8a0d20541"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:22.817549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.817523 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:22.819517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.819475 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115ac86b-e232-4ac4-93ce-72e8a0d20541-kube-api-access-frtsn" pod "115ac86b-e232-4ac4-93ce-72e8a0d20541" (UID: "115ac86b-e232-4ac4-93ce-72e8a0d20541"). InnerVolumeSpecName "kube-api-access-frtsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:22.819689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.819595 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-xks-ca" pod "115ac86b-e232-4ac4-93ce-72e8a0d20541" (UID: "115ac86b-e232-4ac4-93ce-72e8a0d20541"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:22.830963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.830927 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:22.920074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.919935 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115ac86b-e232-4ac4-93ce-72e8a0d20541-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:22.920074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:22.919987 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frtsn\" (UniqueName: \"kubernetes.io/projected/115ac86b-e232-4ac4-93ce-72e8a0d20541-kube-api-access-frtsn\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:23.158113 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.158077 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk"] Oct 30 01:20:23.159639 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:20:23.159607 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189348f5_35d5_4a46_a3ee_c28513b27176.slice/crio-2be2ff0a67512ef214f66740a720528c5ea8d426faae7479e6cf5244e6f5f357 WatchSource:0}: Error finding container 2be2ff0a67512ef214f66740a720528c5ea8d426faae7479e6cf5244e6f5f357: Status 404 returned error can't find the container with id 2be2ff0a67512ef214f66740a720528c5ea8d426faae7479e6cf5244e6f5f357 Oct 30 01:20:23.319059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.319018 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" Oct 30 01:20:23.319059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.319046 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85" event={"ID":"115ac86b-e232-4ac4-93ce-72e8a0d20541","Type":"ContainerDied","Data":"1e464e63c55441bac1622a0dbe0391e260d92883904fc11377e9148c34a77a1e"} Oct 30 01:20:23.319342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.319114 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e464e63c55441bac1622a0dbe0391e260d92883904fc11377e9148c34a77a1e" Oct 30 01:20:23.320887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.320844 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" event={"ID":"189348f5-35d5-4a46-a3ee-c28513b27176","Type":"ContainerStarted","Data":"2be2ff0a67512ef214f66740a720528c5ea8d426faae7479e6cf5244e6f5f357"} Oct 30 01:20:23.377185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.377147 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:23.492216 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.492105 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:23.591621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.567701 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx"] Oct 30 01:20:23.591621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.576173 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv"] Oct 30 01:20:23.623979 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.623935 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28"] Oct 30 01:20:23.674101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.674047 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:23.853333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.853298 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:20:23.955916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.955878 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-data\") pod \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " Oct 30 01:20:23.956136 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.955937 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-certs\") pod \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " Oct 30 01:20:23.956136 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.955976 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istiod-ca-cert\") pod \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " Oct 30 01:20:23.956136 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.956105 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-token\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-token\") pod \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " Oct 30 01:20:23.956357 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.956176 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-socket\") pod \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " Oct 30 01:20:23.956357 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.956210 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-data" pod "28e39131-678d-4eb0-9c12-f9c1e768c3bb" (UID: "28e39131-678d-4eb0-9c12-f9c1e768c3bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:23.956357 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.956214 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-kube-api-access-cq7f2\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-kube-api-access-cq7f2\") pod \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " Oct 30 01:20:23.956357 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.956299 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-envoy\") pod \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " Oct 30 01:20:23.956357 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.956331 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-certs" pod "28e39131-678d-4eb0-9c12-f9c1e768c3bb" (UID: "28e39131-678d-4eb0-9c12-f9c1e768c3bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:23.956653 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.956341 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-podinfo\") pod \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " Oct 30 01:20:23.956653 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.956464 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istiod-ca-cert" pod "28e39131-678d-4eb0-9c12-f9c1e768c3bb" (UID: "28e39131-678d-4eb0-9c12-f9c1e768c3bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:23.956653 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.956623 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-credential-socket\") pod \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\" (UID: \"28e39131-678d-4eb0-9c12-f9c1e768c3bb\") " Oct 30 01:20:23.957481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.957451 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-socket" pod "28e39131-678d-4eb0-9c12-f9c1e768c3bb" (UID: "28e39131-678d-4eb0-9c12-f9c1e768c3bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:23.957616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.957561 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:23.957616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.957574 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-credential-socket" pod "28e39131-678d-4eb0-9c12-f9c1e768c3bb" (UID: "28e39131-678d-4eb0-9c12-f9c1e768c3bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:23.957616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.957586 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:23.958006 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.957623 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:23.970803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.963639 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-podinfo" pod "28e39131-678d-4eb0-9c12-f9c1e768c3bb" (UID: "28e39131-678d-4eb0-9c12-f9c1e768c3bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:20:23.970803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.963649 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-envoy" pod "28e39131-678d-4eb0-9c12-f9c1e768c3bb" (UID: "28e39131-678d-4eb0-9c12-f9c1e768c3bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:23.970803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.963701 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-token" pod "28e39131-678d-4eb0-9c12-f9c1e768c3bb" (UID: "28e39131-678d-4eb0-9c12-f9c1e768c3bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:23.970803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:23.965292 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-kube-api-access-cq7f2" pod "28e39131-678d-4eb0-9c12-f9c1e768c3bb" (UID: "28e39131-678d-4eb0-9c12-f9c1e768c3bb"). InnerVolumeSpecName "kube-api-access-cq7f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:24.058976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.058935 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:24.058976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.058975 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq7f2\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-kube-api-access-cq7f2\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:24.059244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.058991 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:24.059244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.059004 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:24.059244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.059015 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e39131-678d-4eb0-9c12-f9c1e768c3bb-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:24.059244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.059028 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e39131-678d-4eb0-9c12-f9c1e768c3bb-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:24.331331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.331284 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" event={"ID":"189348f5-35d5-4a46-a3ee-c28513b27176","Type":"ContainerStarted","Data":"0935717c22c97b89fe12b67eb453edc61916acb6473475661fb6921ad8e157d1"} Oct 30 01:20:24.332909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.332877 4517 generic.go:356] "Generic (PLEG): container finished" podID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerID="d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5" exitCode=0 Oct 30 01:20:24.333064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.332917 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" Oct 30 01:20:24.333064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.332963 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" event={"ID":"28e39131-678d-4eb0-9c12-f9c1e768c3bb","Type":"ContainerDied","Data":"d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5"} Oct 30 01:20:24.333064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.333021 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7" event={"ID":"28e39131-678d-4eb0-9c12-f9c1e768c3bb","Type":"ContainerDied","Data":"4c71eb0eb7b73ab2fce0b58448f822c3c54483129d18a91fca296a451dec4c16"} Oct 30 01:20:24.333064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.333050 4517 scope.go:122] "RemoveContainer" containerID="d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5" Oct 30 01:20:24.348489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.348460 4517 scope.go:122] "RemoveContainer" containerID="d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5" Oct 30 01:20:24.348940 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:20:24.348916 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5\": container with ID starting with d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5 not found: ID does not exist" containerID="d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5" Oct 30 01:20:24.349017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.348951 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5"} err="failed to get container status \"d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5\": rpc error: code = NotFound desc = could not find container \"d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5\": container with ID starting with d17793fa90bae58c43b222156f873b17a72927a5eab5141cb229787fdd3ad6d5 not found: ID does not exist" Oct 30 01:20:24.370088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.370027 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" podStartSLOduration=2.370011051 podStartE2EDuration="2.370011051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:24.36720711 +0000 UTC m=+2762.279334264" watchObservedRunningTime="2026-10-30 01:20:24.370011051 +0000 UTC m=+2762.282138191" Oct 30 01:20:24.406600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.406552 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7"] Oct 30 01:20:24.411147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:24.411109 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-gfqf7"] Oct 30 01:20:25.276655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:25.276615 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:25.373527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:25.372897 4517 generic.go:356] "Generic (PLEG): container finished" podID="189348f5-35d5-4a46-a3ee-c28513b27176" containerID="0935717c22c97b89fe12b67eb453edc61916acb6473475661fb6921ad8e157d1" exitCode=0 Oct 30 01:20:25.373527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:25.373052 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" event={"ID":"189348f5-35d5-4a46-a3ee-c28513b27176","Type":"ContainerDied","Data":"0935717c22c97b89fe12b67eb453edc61916acb6473475661fb6921ad8e157d1"} Oct 30 01:20:26.236749 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.236697 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" path="/var/lib/kubelet/pods/28e39131-678d-4eb0-9c12-f9c1e768c3bb/volumes" Oct 30 01:20:26.296135 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.296093 4517 scope.go:122] "RemoveContainer" containerID="aaefc81f7f4dec91bd35ea8931ddfa86797db35acc574bf73986edc93ed0c3d5" Oct 30 01:20:26.313811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.313782 4517 scope.go:122] "RemoveContainer" containerID="91f433a2cea2af8a0b34c6cec87dbdad6e0787b4150d96344b2a160fcb01b1ba" Oct 30 01:20:26.324653 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.324628 4517 scope.go:122] "RemoveContainer" containerID="ba4e9ac6b3c6ac7d2944fb4a41e84c0906ceac247f707a90e55fc416b98dab70" Oct 30 01:20:26.335983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.335957 4517 scope.go:122] "RemoveContainer" containerID="a60558906294521250a268f2687fe2e7f200a163812fea432692f563ac62c0c0" Oct 30 01:20:26.592326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.592296 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:26.703407 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.703364 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-xks-ca\") pod \"189348f5-35d5-4a46-a3ee-c28513b27176\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " Oct 30 01:20:26.703664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.703648 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189348f5-35d5-4a46-a3ee-c28513b27176-kube-api-access-qvp68\" (UniqueName: \"kubernetes.io/projected/189348f5-35d5-4a46-a3ee-c28513b27176-kube-api-access-qvp68\") pod \"189348f5-35d5-4a46-a3ee-c28513b27176\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " Oct 30 01:20:26.703742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.703688 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-openshift-ca\") pod \"189348f5-35d5-4a46-a3ee-c28513b27176\" (UID: \"189348f5-35d5-4a46-a3ee-c28513b27176\") " Oct 30 01:20:26.704177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.704141 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-openshift-ca" pod "189348f5-35d5-4a46-a3ee-c28513b27176" (UID: "189348f5-35d5-4a46-a3ee-c28513b27176"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:26.704467 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.704377 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:26.705983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.705944 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-xks-ca" pod "189348f5-35d5-4a46-a3ee-c28513b27176" (UID: "189348f5-35d5-4a46-a3ee-c28513b27176"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:26.706552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.706526 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189348f5-35d5-4a46-a3ee-c28513b27176-kube-api-access-qvp68" pod "189348f5-35d5-4a46-a3ee-c28513b27176" (UID: "189348f5-35d5-4a46-a3ee-c28513b27176"). InnerVolumeSpecName "kube-api-access-qvp68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:26.805819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.805697 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvp68\" (UniqueName: \"kubernetes.io/projected/189348f5-35d5-4a46-a3ee-c28513b27176-kube-api-access-qvp68\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:26.805819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:26.805753 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189348f5-35d5-4a46-a3ee-c28513b27176-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.192760 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:27.192607 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:27.400954 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:27.400920 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" Oct 30 01:20:27.401382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:27.400916 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk" event={"ID":"189348f5-35d5-4a46-a3ee-c28513b27176","Type":"ContainerDied","Data":"2be2ff0a67512ef214f66740a720528c5ea8d426faae7479e6cf5244e6f5f357"} Oct 30 01:20:27.401382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:27.401029 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be2ff0a67512ef214f66740a720528c5ea8d426faae7479e6cf5244e6f5f357" Oct 30 01:20:28.535488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.535454 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:28.772778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.770155 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.962789 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg"] Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.963855 4517 cpu_manager.go:414] "containerMap: removing container" podUID="115ac86b-e232-4ac4-93ce-72e8a0d20541" containerName="revoke-keys" Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.963880 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115ac86b-e232-4ac4-93ce-72e8a0d20541" containerName="revoke-keys" Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.963917 4517 cpu_manager.go:414] "containerMap: removing container" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.963927 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.963953 4517 cpu_manager.go:414] "containerMap: removing container" podUID="189348f5-35d5-4a46-a3ee-c28513b27176" containerName="revoke-keys" Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.963962 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189348f5-35d5-4a46-a3ee-c28513b27176" containerName="revoke-keys" Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.964153 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="115ac86b-e232-4ac4-93ce-72e8a0d20541" containerName="revoke-keys" Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.964172 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e39131-678d-4eb0-9c12-f9c1e768c3bb" containerName="istio-proxy" Oct 30 01:20:28.966793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:28.964186 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="189348f5-35d5-4a46-a3ee-c28513b27176" containerName="revoke-keys" Oct 30 01:20:29.001743 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.001473 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:29.062744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.057102 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-38b47d\"" Oct 30 01:20:29.062744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.057215 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-38b47d-dockercfg-4899d\"" Oct 30 01:20:29.236857 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.236761 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg"] Oct 30 01:20:29.303939 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.303893 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jws4v\" (UniqueName: \"kubernetes.io/projected/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-kube-api-access-jws4v\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg\" (UID: \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:29.304277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.304250 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg\" (UID: \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:29.314869 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.314782 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5"] Oct 30 01:20:29.320592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.320555 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5"] Oct 30 01:20:29.320918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.320826 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:29.406424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.406365 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dbt\" (UniqueName: \"kubernetes.io/projected/2183c7f2-b9bf-4caa-9761-249a6736dc59-kube-api-access-l8dbt\") pod \"payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5\" (UID: \"2183c7f2-b9bf-4caa-9761-249a6736dc59\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:29.406651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.406597 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jws4v\" (UniqueName: \"kubernetes.io/projected/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-kube-api-access-jws4v\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg\" (UID: \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:29.406651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.406643 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg\" (UID: \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:29.406883 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.406864 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183c7f2-b9bf-4caa-9761-249a6736dc59-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5\" (UID: \"2183c7f2-b9bf-4caa-9761-249a6736dc59\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:29.407669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.407642 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg\" (UID: \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:29.432249 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.432213 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jws4v\" (UniqueName: \"kubernetes.io/projected/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-kube-api-access-jws4v\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg\" (UID: \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:29.455055 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.455006 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:29.508294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.508192 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183c7f2-b9bf-4caa-9761-249a6736dc59-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5\" (UID: \"2183c7f2-b9bf-4caa-9761-249a6736dc59\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:29.508494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.508362 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dbt\" (UniqueName: \"kubernetes.io/projected/2183c7f2-b9bf-4caa-9761-249a6736dc59-kube-api-access-l8dbt\") pod \"payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5\" (UID: \"2183c7f2-b9bf-4caa-9761-249a6736dc59\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:29.509085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.509057 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183c7f2-b9bf-4caa-9761-249a6736dc59-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5\" (UID: \"2183c7f2-b9bf-4caa-9761-249a6736dc59\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:29.532911 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.532879 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dbt\" (UniqueName: \"kubernetes.io/projected/2183c7f2-b9bf-4caa-9761-249a6736dc59-kube-api-access-l8dbt\") pod \"payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5\" (UID: \"2183c7f2-b9bf-4caa-9761-249a6736dc59\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:29.578731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.577959 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:29.606733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.604356 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:29.642453 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:29.642413 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:30.158732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:30.158095 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg"] Oct 30 01:20:30.184732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:30.159922 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:30.266073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:30.265093 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5"] Oct 30 01:20:30.281938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:30.278330 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:30.436602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:30.436556 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" event={"ID":"2183c7f2-b9bf-4caa-9761-249a6736dc59","Type":"ContainerStarted","Data":"ffd7d1d56ef7d003f9f2efa544e9a730e41e65067f114e697be8bd8d9b2672f0"} Oct 30 01:20:30.438477 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:30.438443 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" event={"ID":"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1","Type":"ContainerStarted","Data":"dc288a442a45857070e234ca0ae3aabe02cd55f73b1493f204b292d7b9fc8dda"} Oct 30 01:20:30.544561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:30.544522 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:30.761823 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:20:30.761694 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:20:31.449112 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:31.449073 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" event={"ID":"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1","Type":"ContainerStarted","Data":"58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560"} Oct 30 01:20:31.449337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:31.449175 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:31.451185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:31.451155 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" event={"ID":"2183c7f2-b9bf-4caa-9761-249a6736dc59","Type":"ContainerStarted","Data":"7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b"} Oct 30 01:20:31.451323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:31.451284 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:31.486672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:31.486602 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" podStartSLOduration=3.48658427 podStartE2EDuration="3.48658427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:31.4860676 +0000 UTC m=+2769.398194742" watchObservedRunningTime="2026-10-30 01:20:31.48658427 +0000 UTC m=+2769.398711416" Oct 30 01:20:31.528333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:31.528264 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" podStartSLOduration=2.5282478 podStartE2EDuration="2.5282478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:31.523489338 +0000 UTC m=+2769.435616482" watchObservedRunningTime="2026-10-30 01:20:31.5282478 +0000 UTC m=+2769.440374981" Oct 30 01:20:32.175459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:32.175424 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:32.311264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:32.310658 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:32.429615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:32.429527 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:33.316417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:33.316363 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7"] Oct 30 01:20:33.427992 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:33.427912 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:33.588812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:33.581787 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:34.507300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:34.507259 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:36.170513 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:36.170473 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:37.827946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:37.827913 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:38.484389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.484332 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" podUID="4ea1f145-1eb3-49a4-8658-b32e7b228ebb" containerName="payload-pre-processing" containerID="cri-o://4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674" gracePeriod=25 Oct 30 01:20:38.640099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.640035 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" containerID="cri-o://ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb" gracePeriod=45 Oct 30 01:20:38.656888 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.656822 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" containerID="cri-o://73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7" gracePeriod=45 Oct 30 01:20:38.660590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.660354 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" containerID="cri-o://32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b" gracePeriod=45 Oct 30 01:20:38.898804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.898265 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:20:38.919570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.919207 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused" Oct 30 01:20:38.960157 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.960102 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused" Oct 30 01:20:38.987359 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.987325 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-plugins-config-volume\") pod \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\" (UID: \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\") " Oct 30 01:20:38.987521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.987394 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-kube-api-access-fhjmf\" (UniqueName: \"kubernetes.io/projected/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-kube-api-access-fhjmf\") pod \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\" (UID: \"4ea1f145-1eb3-49a4-8658-b32e7b228ebb\") " Oct 30 01:20:38.989116 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:38.989073 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-plugins-config-volume" pod "4ea1f145-1eb3-49a4-8658-b32e7b228ebb" (UID: "4ea1f145-1eb3-49a4-8658-b32e7b228ebb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:39.028001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.027958 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-kube-api-access-fhjmf" pod "4ea1f145-1eb3-49a4-8658-b32e7b228ebb" (UID: "4ea1f145-1eb3-49a4-8658-b32e7b228ebb"). InnerVolumeSpecName "kube-api-access-fhjmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:39.044012 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.043971 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 01:20:39.089306 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.089264 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-model-cache\") pod \"9c3e0328-65f0-4890-b71d-242bea9998c8\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " Oct 30 01:20:39.089501 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.089439 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-tmp-dir\") pod \"9c3e0328-65f0-4890-b71d-242bea9998c8\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " Oct 30 01:20:39.089597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.089586 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-home\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-home\") pod \"9c3e0328-65f0-4890-b71d-242bea9998c8\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " Oct 30 01:20:39.089701 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.089681 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3e0328-65f0-4890-b71d-242bea9998c8-kube-api-access-mtnzd\" (UniqueName: \"kubernetes.io/projected/9c3e0328-65f0-4890-b71d-242bea9998c8-kube-api-access-mtnzd\") pod \"9c3e0328-65f0-4890-b71d-242bea9998c8\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " Oct 30 01:20:39.089805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.089768 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-dshm\") pod \"9c3e0328-65f0-4890-b71d-242bea9998c8\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " Oct 30 01:20:39.089805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.089795 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c3e0328-65f0-4890-b71d-242bea9998c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3e0328-65f0-4890-b71d-242bea9998c8-tls-certs\") pod \"9c3e0328-65f0-4890-b71d-242bea9998c8\" (UID: \"9c3e0328-65f0-4890-b71d-242bea9998c8\") " Oct 30 01:20:39.091469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.091036 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.091469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.091080 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjmf\" (UniqueName: \"kubernetes.io/projected/4ea1f145-1eb3-49a4-8658-b32e7b228ebb-kube-api-access-fhjmf\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.092008 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.091978 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-home" pod "9c3e0328-65f0-4890-b71d-242bea9998c8" (UID: "9c3e0328-65f0-4890-b71d-242bea9998c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.092475 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.092334 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-model-cache" pod "9c3e0328-65f0-4890-b71d-242bea9998c8" (UID: "9c3e0328-65f0-4890-b71d-242bea9998c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.092556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.092518 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-tmp-dir" pod "9c3e0328-65f0-4890-b71d-242bea9998c8" (UID: "9c3e0328-65f0-4890-b71d-242bea9998c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.094932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.094591 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3e0328-65f0-4890-b71d-242bea9998c8-tls-certs" pod "9c3e0328-65f0-4890-b71d-242bea9998c8" (UID: "9c3e0328-65f0-4890-b71d-242bea9998c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:39.097494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.097443 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-dshm" pod "9c3e0328-65f0-4890-b71d-242bea9998c8" (UID: "9c3e0328-65f0-4890-b71d-242bea9998c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.100345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.100298 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3e0328-65f0-4890-b71d-242bea9998c8-kube-api-access-mtnzd" pod "9c3e0328-65f0-4890-b71d-242bea9998c8" (UID: "9c3e0328-65f0-4890-b71d-242bea9998c8"). InnerVolumeSpecName "kube-api-access-mtnzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:39.109238 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.109197 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 01:20:39.139324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.139259 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.191600 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af0882d-f4f4-4613-b90f-42ef0b01efc6-kube-api-access-wmm2k\" (UniqueName: \"kubernetes.io/projected/1af0882d-f4f4-4613-b90f-42ef0b01efc6-kube-api-access-wmm2k\") pod \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.191756 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-home\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-home\") pod \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.191879 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-home\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-home\") pod \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.191918 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-model-cache\") pod \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.191998 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-kube-api-access-67q8b\" (UniqueName: \"kubernetes.io/projected/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-kube-api-access-67q8b\") pod \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.192059 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-dshm\") pod \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.192111 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tls-certs\") pod \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.192142 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tmp-dir\") pod \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.192175 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tmp-dir\") pod \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.192279 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tls-certs\") pod \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\" (UID: \"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.192311 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-model-cache\") pod \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " Oct 30 01:20:39.192733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.192348 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-dshm\") pod \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\" (UID: \"1af0882d-f4f4-4613-b90f-42ef0b01efc6\") " Oct 30 01:20:39.193502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.193050 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.193502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.193091 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3e0328-65f0-4890-b71d-242bea9998c8-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.193502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.193110 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.193502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.193124 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.193502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.193138 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c3e0328-65f0-4890-b71d-242bea9998c8-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.193502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.193151 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtnzd\" (UniqueName: \"kubernetes.io/projected/9c3e0328-65f0-4890-b71d-242bea9998c8-kube-api-access-mtnzd\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.194168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.193858 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-home" pod "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" (UID: "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.195688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.194846 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tmp-dir" pod "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" (UID: "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.195688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.195132 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-home" pod "1af0882d-f4f4-4613-b90f-42ef0b01efc6" (UID: "1af0882d-f4f4-4613-b90f-42ef0b01efc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.195688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.195148 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-model-cache" pod "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" (UID: "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.195688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.195365 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tmp-dir" pod "1af0882d-f4f4-4613-b90f-42ef0b01efc6" (UID: "1af0882d-f4f4-4613-b90f-42ef0b01efc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.195688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.195640 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-model-cache" pod "1af0882d-f4f4-4613-b90f-42ef0b01efc6" (UID: "1af0882d-f4f4-4613-b90f-42ef0b01efc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.198458 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.198406 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tls-certs" pod "1af0882d-f4f4-4613-b90f-42ef0b01efc6" (UID: "1af0882d-f4f4-4613-b90f-42ef0b01efc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:39.201350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.201310 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-kube-api-access-67q8b" pod "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" (UID: "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9"). InnerVolumeSpecName "kube-api-access-67q8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:39.201675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.201628 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af0882d-f4f4-4613-b90f-42ef0b01efc6-kube-api-access-wmm2k" pod "1af0882d-f4f4-4613-b90f-42ef0b01efc6" (UID: "1af0882d-f4f4-4613-b90f-42ef0b01efc6"). InnerVolumeSpecName "kube-api-access-wmm2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:39.203244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.203185 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-dshm" pod "1af0882d-f4f4-4613-b90f-42ef0b01efc6" (UID: "1af0882d-f4f4-4613-b90f-42ef0b01efc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.205216 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.205170 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tls-certs" pod "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" (UID: "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:39.206184 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.206138 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-dshm" pod "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" (UID: "89ae0bf1-3213-46b0-8d0c-58ce5fb889a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:39.294651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294574 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmm2k\" (UniqueName: \"kubernetes.io/projected/1af0882d-f4f4-4613-b90f-42ef0b01efc6-kube-api-access-wmm2k\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.294651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294612 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.294651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294628 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.294651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294641 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.294651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294656 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67q8b\" (UniqueName: \"kubernetes.io/projected/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-kube-api-access-67q8b\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.295090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294668 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.295090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294683 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.295090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294696 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.295090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294727 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.295090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294739 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.295090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294750 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.295090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.294761 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af0882d-f4f4-4613-b90f-42ef0b01efc6-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:39.550860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.550771 4517 generic.go:356] "Generic (PLEG): container finished" podID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerID="73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7" exitCode=0 Oct 30 01:20:39.550860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.550821 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" Oct 30 01:20:39.551127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.550855 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" event={"ID":"9c3e0328-65f0-4890-b71d-242bea9998c8","Type":"ContainerDied","Data":"73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7"} Oct 30 01:20:39.551127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.550909 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx" event={"ID":"9c3e0328-65f0-4890-b71d-242bea9998c8","Type":"ContainerDied","Data":"7e9f7f9506dc064f024571696558dee07f15feb7a7efce095a77bb7f245d48a7"} Oct 30 01:20:39.551127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.550924 4517 scope.go:122] "RemoveContainer" containerID="73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7" Oct 30 01:20:39.552776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.552748 4517 generic.go:356] "Generic (PLEG): container finished" podID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerID="32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b" exitCode=0 Oct 30 01:20:39.552917 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.552883 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" event={"ID":"1af0882d-f4f4-4613-b90f-42ef0b01efc6","Type":"ContainerDied","Data":"32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b"} Oct 30 01:20:39.552917 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.552901 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" Oct 30 01:20:39.553019 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.552933 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28" event={"ID":"1af0882d-f4f4-4613-b90f-42ef0b01efc6","Type":"ContainerDied","Data":"3619e45bd90c948e89bf45325ea24b3a77e97f3881ad3c466d1e18313f5413bc"} Oct 30 01:20:39.554633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.554603 4517 generic.go:356] "Generic (PLEG): container finished" podID="4ea1f145-1eb3-49a4-8658-b32e7b228ebb" containerID="4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674" exitCode=0 Oct 30 01:20:39.554828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.554653 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" Oct 30 01:20:39.554828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.554699 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" event={"ID":"4ea1f145-1eb3-49a4-8658-b32e7b228ebb","Type":"ContainerDied","Data":"4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674"} Oct 30 01:20:39.554828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.554793 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7" event={"ID":"4ea1f145-1eb3-49a4-8658-b32e7b228ebb","Type":"ContainerDied","Data":"3f699f40baaf3e529edb90c20e9384228acb9236a5cc21e8cbccfe04b797890f"} Oct 30 01:20:39.556639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.556607 4517 generic.go:356] "Generic (PLEG): container finished" podID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerID="ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb" exitCode=0 Oct 30 01:20:39.556820 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.556649 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" Oct 30 01:20:39.556820 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.556750 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" event={"ID":"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9","Type":"ContainerDied","Data":"ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb"} Oct 30 01:20:39.556820 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.556791 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv" event={"ID":"89ae0bf1-3213-46b0-8d0c-58ce5fb889a9","Type":"ContainerDied","Data":"bbaac542aa84471f3553825f630274d4df32431e683287a1cc4bc30201561c71"} Oct 30 01:20:39.571998 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.571974 4517 scope.go:122] "RemoveContainer" containerID="73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7" Oct 30 01:20:39.572445 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:20:39.572406 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7\": container with ID starting with 73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7 not found: ID does not exist" containerID="73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7" Oct 30 01:20:39.572578 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.572463 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7"} err="failed to get container status \"73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7\": rpc error: code = NotFound desc = could not find container \"73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7\": container with ID starting with 73139f7ef423645a4fece24d113ac0f0c5b865c9a46a83f7e02dfa6c3baf37a7 not found: ID does not exist" Oct 30 01:20:39.572578 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.572487 4517 scope.go:122] "RemoveContainer" containerID="32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b" Oct 30 01:20:39.587030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.587007 4517 scope.go:122] "RemoveContainer" containerID="32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b" Oct 30 01:20:39.587399 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:20:39.587377 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b\": container with ID starting with 32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b not found: ID does not exist" containerID="32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b" Oct 30 01:20:39.587491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.587407 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b"} err="failed to get container status \"32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b\": rpc error: code = NotFound desc = could not find container \"32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b\": container with ID starting with 32db23d67a898b4ade156bae309a10faf815ee62da7aa4e94bdf05e81230719b not found: ID does not exist" Oct 30 01:20:39.587491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.587424 4517 scope.go:122] "RemoveContainer" containerID="4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674" Oct 30 01:20:39.599872 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.599840 4517 scope.go:122] "RemoveContainer" containerID="4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674" Oct 30 01:20:39.600281 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:20:39.600259 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674\": container with ID starting with 4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674 not found: ID does not exist" containerID="4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674" Oct 30 01:20:39.600389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.600294 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674"} err="failed to get container status \"4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674\": rpc error: code = NotFound desc = could not find container \"4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674\": container with ID starting with 4d1478eca3df96169c544551da07c37247db691099f171f62b40bcb9284cc674 not found: ID does not exist" Oct 30 01:20:39.600389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.600315 4517 scope.go:122] "RemoveContainer" containerID="ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb" Oct 30 01:20:39.613959 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.613931 4517 scope.go:122] "RemoveContainer" containerID="ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb" Oct 30 01:20:39.614360 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:20:39.614329 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb\": container with ID starting with ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb not found: ID does not exist" containerID="ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb" Oct 30 01:20:39.614479 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.614368 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb"} err="failed to get container status \"ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb\": rpc error: code = NotFound desc = could not find container \"ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb\": container with ID starting with ed725c1eca96933d52c8c9f62c26bcd4896197bffe8b9f9703d95d981f0913fb not found: ID does not exist" Oct 30 01:20:39.628223 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.628183 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx"] Oct 30 01:20:39.638539 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.638499 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-2-w3-b3593a-kserve-6c665f6d64-6s5jx"] Oct 30 01:20:39.677304 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.677262 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28"] Oct 30 01:20:39.683057 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.683022 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/premium-w3-b3593a-kserve-6c9db7bd68-xwp28"] Oct 30 01:20:39.696361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.696326 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:39.715271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.715228 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv"] Oct 30 01:20:39.722286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.722246 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/distinct-w3-b3593a-kserve-7cc6c5f487-pd4rv"] Oct 30 01:20:39.805772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.805591 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7"] Oct 30 01:20:39.814105 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:39.814064 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a42e5-5f495b4bf9-bpmp7"] Oct 30 01:20:40.237021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:40.236932 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" path="/var/lib/kubelet/pods/1af0882d-f4f4-4613-b90f-42ef0b01efc6/volumes" Oct 30 01:20:40.237607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:40.237589 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea1f145-1eb3-49a4-8658-b32e7b228ebb" path="/var/lib/kubelet/pods/4ea1f145-1eb3-49a4-8658-b32e7b228ebb/volumes" Oct 30 01:20:40.238146 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:40.238128 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" path="/var/lib/kubelet/pods/89ae0bf1-3213-46b0-8d0c-58ce5fb889a9/volumes" Oct 30 01:20:40.238568 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:40.238554 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" path="/var/lib/kubelet/pods/9c3e0328-65f0-4890-b71d-242bea9998c8/volumes" Oct 30 01:20:41.166355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:41.166262 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:41.922016 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:41.921977 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:42.042366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:42.042334 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:42.049047 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:42.049009 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk"] Oct 30 01:20:42.058907 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:42.058868 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a42e5-l54fk"] Oct 30 01:20:42.163233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:42.163199 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:42.240268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:42.240159 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189348f5-35d5-4a46-a3ee-c28513b27176" path="/var/lib/kubelet/pods/189348f5-35d5-4a46-a3ee-c28513b27176/volumes" Oct 30 01:20:42.463945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:42.463905 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:20:42.465275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:42.465250 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:20:42.541366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:42.541326 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:43.927566 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:43.927532 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:45.611872 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:45.611835 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:47.063190 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.063152 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:47.788559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.787823 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9"] Oct 30 01:20:47.789093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.788826 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" Oct 30 01:20:47.789093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.788853 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" Oct 30 01:20:47.789093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.788915 4517 cpu_manager.go:414] "containerMap: removing container" podUID="4ea1f145-1eb3-49a4-8658-b32e7b228ebb" containerName="payload-pre-processing" Oct 30 01:20:47.789093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.788923 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea1f145-1eb3-49a4-8658-b32e7b228ebb" containerName="payload-pre-processing" Oct 30 01:20:47.789093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.788939 4517 cpu_manager.go:414] "containerMap: removing container" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" Oct 30 01:20:47.789093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.788946 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" Oct 30 01:20:47.789093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.788963 4517 cpu_manager.go:414] "containerMap: removing container" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" Oct 30 01:20:47.789093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.788970 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" Oct 30 01:20:47.789618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.789133 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea1f145-1eb3-49a4-8658-b32e7b228ebb" containerName="payload-pre-processing" Oct 30 01:20:47.789618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.789146 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3e0328-65f0-4890-b71d-242bea9998c8" containerName="main" Oct 30 01:20:47.789618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.789158 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ae0bf1-3213-46b0-8d0c-58ce5fb889a9" containerName="main" Oct 30 01:20:47.789618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.789172 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af0882d-f4f4-4613-b90f-42ef0b01efc6" containerName="main" Oct 30 01:20:47.795510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.795474 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927"] Oct 30 01:20:47.801729 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.801416 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:47.803802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.803768 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9"] Oct 30 01:20:47.809121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.808896 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:47.811510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.811060 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927"] Oct 30 01:20:47.811510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.811344 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:47.824449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.824391 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:20:47.825448 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.825428 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:20:47.906987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.906949 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f"] Oct 30 01:20:47.912482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.912447 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6c4f6f-gl78f"] Oct 30 01:20:47.954353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.954316 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:47.985731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.975383 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e2a5697-kcnb9\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:47.986613 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.986578 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e2a5697-kcnb9\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:47.986841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.986819 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ceed4a7-q8927\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:47.998730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.987215 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ceed4a7-q8927\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:47.999919 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:47.999886 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnxj\" (UniqueName: \"kubernetes.io/projected/ece50684-1c45-48f6-b3a7-23f2aa6e888d-kube-api-access-tdnxj\") pod \"maas-api-revoke-sub-7ceed4a7-q8927\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:48.000270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.000249 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9sl\" (UniqueName: \"kubernetes.io/projected/d17b1bc9-1b6b-499b-857c-994bb6283d0a-kube-api-access-9t9sl\") pod \"maas-api-revoke-sub-5e2a5697-kcnb9\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:48.102310 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.102198 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ceed4a7-q8927\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:48.102310 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.102259 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnxj\" (UniqueName: \"kubernetes.io/projected/ece50684-1c45-48f6-b3a7-23f2aa6e888d-kube-api-access-tdnxj\") pod \"maas-api-revoke-sub-7ceed4a7-q8927\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:48.102874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.102403 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9sl\" (UniqueName: \"kubernetes.io/projected/d17b1bc9-1b6b-499b-857c-994bb6283d0a-kube-api-access-9t9sl\") pod \"maas-api-revoke-sub-5e2a5697-kcnb9\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:48.102874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.102446 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e2a5697-kcnb9\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:48.102874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.102495 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e2a5697-kcnb9\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:48.102874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.102527 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ceed4a7-q8927\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:48.103250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.103227 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e2a5697-kcnb9\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:48.103641 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.103619 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ceed4a7-q8927\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:48.105503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.105474 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ceed4a7-q8927\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:48.105812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.105789 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e2a5697-kcnb9\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:48.143805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.143515 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnxj\" (UniqueName: \"kubernetes.io/projected/ece50684-1c45-48f6-b3a7-23f2aa6e888d-kube-api-access-tdnxj\") pod \"maas-api-revoke-sub-7ceed4a7-q8927\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:48.147108 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.145434 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9sl\" (UniqueName: \"kubernetes.io/projected/d17b1bc9-1b6b-499b-857c-994bb6283d0a-kube-api-access-9t9sl\") pod \"maas-api-revoke-sub-5e2a5697-kcnb9\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:48.149335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.149299 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:48.149862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.149838 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:48.237916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.237880 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717e8a03-586b-41c3-8630-bc67ff068221" path="/var/lib/kubelet/pods/717e8a03-586b-41c3-8630-bc67ff068221/volumes" Oct 30 01:20:48.375590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.375493 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:48.439984 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.439942 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:48.520608 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.519432 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927"] Oct 30 01:20:48.525588 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:20:48.525529 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece50684_1c45_48f6_b3a7_23f2aa6e888d.slice/crio-97887f5a2fcbecc258573d1337739f7749d63d18433d6fc4b9cd74f71fe0049f WatchSource:0}: Error finding container 97887f5a2fcbecc258573d1337739f7749d63d18433d6fc4b9cd74f71fe0049f: Status 404 returned error can't find the container with id 97887f5a2fcbecc258573d1337739f7749d63d18433d6fc4b9cd74f71fe0049f Oct 30 01:20:48.708999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.708891 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5_2183c7f2-b9bf-4caa-9761-249a6736dc59/payload-processing/0.log" Oct 30 01:20:48.784581 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.784547 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9"] Oct 30 01:20:48.786260 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:20:48.786220 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17b1bc9_1b6b_499b_857c_994bb6283d0a.slice/crio-1fab56117eb220c83859df19f2797378502cae2bc1e014fd29c52176a966ee25 WatchSource:0}: Error finding container 1fab56117eb220c83859df19f2797378502cae2bc1e014fd29c52176a966ee25: Status 404 returned error can't find the container with id 1fab56117eb220c83859df19f2797378502cae2bc1e014fd29c52176a966ee25 Oct 30 01:20:48.828779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.828739 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" event={"ID":"ece50684-1c45-48f6-b3a7-23f2aa6e888d","Type":"ContainerStarted","Data":"97887f5a2fcbecc258573d1337739f7749d63d18433d6fc4b9cd74f71fe0049f"} Oct 30 01:20:48.830301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:48.830261 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" event={"ID":"d17b1bc9-1b6b-499b-857c-994bb6283d0a","Type":"ContainerStarted","Data":"1fab56117eb220c83859df19f2797378502cae2bc1e014fd29c52176a966ee25"} Oct 30 01:20:49.732748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:49.732694 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:49.846594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:49.846527 4517 generic.go:356] "Generic (PLEG): container finished" podID="ece50684-1c45-48f6-b3a7-23f2aa6e888d" containerID="402e940909855b4cdbe8d4787bc9f6e5294645169299a475cc23a60a554196c8" exitCode=0 Oct 30 01:20:49.846890 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:49.846855 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" event={"ID":"ece50684-1c45-48f6-b3a7-23f2aa6e888d","Type":"ContainerDied","Data":"402e940909855b4cdbe8d4787bc9f6e5294645169299a475cc23a60a554196c8"} Oct 30 01:20:49.849608 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:49.849568 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" event={"ID":"d17b1bc9-1b6b-499b-857c-994bb6283d0a","Type":"ContainerStarted","Data":"03ee718ef0c7b17071fd3dca33b2ca536466799280e8b24b32faa02af735c17b"} Oct 30 01:20:49.853872 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:49.853841 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:49.934850 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:49.934742 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" podStartSLOduration=2.934725947 podStartE2EDuration="2.934725947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:49.930776294 +0000 UTC m=+2787.842903432" watchObservedRunningTime="2026-10-30 01:20:49.934725947 +0000 UTC m=+2787.846853084" Oct 30 01:20:50.082744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:50.070905 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8"] Oct 30 01:20:50.082744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:50.071180 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" containerID="cri-o://c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f" gracePeriod=30 Oct 30 01:20:50.115561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:50.115509 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:50.860432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:50.860392 4517 generic.go:356] "Generic (PLEG): container finished" podID="d17b1bc9-1b6b-499b-857c-994bb6283d0a" containerID="03ee718ef0c7b17071fd3dca33b2ca536466799280e8b24b32faa02af735c17b" exitCode=0 Oct 30 01:20:50.860958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:50.860479 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" event={"ID":"d17b1bc9-1b6b-499b-857c-994bb6283d0a","Type":"ContainerDied","Data":"03ee718ef0c7b17071fd3dca33b2ca536466799280e8b24b32faa02af735c17b"} Oct 30 01:20:50.951378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:50.951340 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:51.081022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.080995 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:51.189066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.188966 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-openshift-ca\") pod \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " Oct 30 01:20:51.189245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.189091 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-xks-ca\") pod \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " Oct 30 01:20:51.189245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.189202 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece50684-1c45-48f6-b3a7-23f2aa6e888d-kube-api-access-tdnxj\" (UniqueName: \"kubernetes.io/projected/ece50684-1c45-48f6-b3a7-23f2aa6e888d-kube-api-access-tdnxj\") pod \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\" (UID: \"ece50684-1c45-48f6-b3a7-23f2aa6e888d\") " Oct 30 01:20:51.189472 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.189444 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-openshift-ca" pod "ece50684-1c45-48f6-b3a7-23f2aa6e888d" (UID: "ece50684-1c45-48f6-b3a7-23f2aa6e888d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:51.189982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.189911 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.191901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.191866 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece50684-1c45-48f6-b3a7-23f2aa6e888d-kube-api-access-tdnxj" pod "ece50684-1c45-48f6-b3a7-23f2aa6e888d" (UID: "ece50684-1c45-48f6-b3a7-23f2aa6e888d"). InnerVolumeSpecName "kube-api-access-tdnxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:51.199376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.199333 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-xks-ca" pod "ece50684-1c45-48f6-b3a7-23f2aa6e888d" (UID: "ece50684-1c45-48f6-b3a7-23f2aa6e888d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:51.291752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.291607 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece50684-1c45-48f6-b3a7-23f2aa6e888d-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.291752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.291645 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdnxj\" (UniqueName: \"kubernetes.io/projected/ece50684-1c45-48f6-b3a7-23f2aa6e888d-kube-api-access-tdnxj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.872626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.872596 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" Oct 30 01:20:51.873224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.872632 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927" event={"ID":"ece50684-1c45-48f6-b3a7-23f2aa6e888d","Type":"ContainerDied","Data":"97887f5a2fcbecc258573d1337739f7749d63d18433d6fc4b9cd74f71fe0049f"} Oct 30 01:20:51.873224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:51.872746 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97887f5a2fcbecc258573d1337739f7749d63d18433d6fc4b9cd74f71fe0049f" Oct 30 01:20:52.058808 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.058774 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:52.204225 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.204107 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-xks-ca\") pod \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " Oct 30 01:20:52.204440 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.204417 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17b1bc9-1b6b-499b-857c-994bb6283d0a-kube-api-access-9t9sl\" (UniqueName: \"kubernetes.io/projected/d17b1bc9-1b6b-499b-857c-994bb6283d0a-kube-api-access-9t9sl\") pod \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " Oct 30 01:20:52.204608 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.204590 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-openshift-ca\") pod \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\" (UID: \"d17b1bc9-1b6b-499b-857c-994bb6283d0a\") " Oct 30 01:20:52.205055 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.204982 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-openshift-ca" pod "d17b1bc9-1b6b-499b-857c-994bb6283d0a" (UID: "d17b1bc9-1b6b-499b-857c-994bb6283d0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:52.205319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.205299 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:52.206606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.206576 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-xks-ca" pod "d17b1bc9-1b6b-499b-857c-994bb6283d0a" (UID: "d17b1bc9-1b6b-499b-857c-994bb6283d0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:52.207288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.207256 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17b1bc9-1b6b-499b-857c-994bb6283d0a-kube-api-access-9t9sl" pod "d17b1bc9-1b6b-499b-857c-994bb6283d0a" (UID: "d17b1bc9-1b6b-499b-857c-994bb6283d0a"). InnerVolumeSpecName "kube-api-access-9t9sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:52.306994 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.306958 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t9sl\" (UniqueName: \"kubernetes.io/projected/d17b1bc9-1b6b-499b-857c-994bb6283d0a-kube-api-access-9t9sl\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:52.306994 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.306995 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17b1bc9-1b6b-499b-857c-994bb6283d0a-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:52.758739 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.754754 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:52.884238 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.884206 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" Oct 30 01:20:52.884696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.884201 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9" event={"ID":"d17b1bc9-1b6b-499b-857c-994bb6283d0a","Type":"ContainerDied","Data":"1fab56117eb220c83859df19f2797378502cae2bc1e014fd29c52176a966ee25"} Oct 30 01:20:52.884696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:52.884342 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fab56117eb220c83859df19f2797378502cae2bc1e014fd29c52176a966ee25" Oct 30 01:20:54.526514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:54.526481 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:55.048617 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.048585 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:55.163389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.163350 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:55.339731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.338759 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:20:55.406498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.406466 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:55.458733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.458403 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-podinfo\") pod \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " Oct 30 01:20:55.458733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.458469 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-token\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-token\") pod \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " Oct 30 01:20:55.458733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.458543 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-kube-api-access-2g7tw\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-kube-api-access-2g7tw\") pod \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " Oct 30 01:20:55.458733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.458573 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-socket\") pod \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " Oct 30 01:20:55.458733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.458661 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-data\") pod \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " Oct 30 01:20:55.477737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.475890 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-socket" pod "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" (UID: "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:55.500000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.489620 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-credential-socket\") pod \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " Oct 30 01:20:55.500000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.489695 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-envoy\") pod \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " Oct 30 01:20:55.500000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.489806 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istiod-ca-cert\") pod \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " Oct 30 01:20:55.500000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.489876 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-certs\") pod \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\" (UID: \"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2\") " Oct 30 01:20:55.500000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.491041 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.500000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.491403 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-certs" pod "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" (UID: "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:55.500000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.491667 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-credential-socket" pod "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" (UID: "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:55.500000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.492940 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-data" pod "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" (UID: "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:55.500000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.493306 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istiod-ca-cert" pod "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" (UID: "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:55.678049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.623369 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.678049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.623416 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.678049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.623431 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.678049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.623444 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.717035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.716680 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-kube-api-access-2g7tw" pod "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" (UID: "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2"). InnerVolumeSpecName "kube-api-access-2g7tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:55.725964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.725271 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g7tw\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-kube-api-access-2g7tw\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.727867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.727815 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-token" pod "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" (UID: "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:55.728204 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.728173 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-envoy" pod "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" (UID: "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:55.728870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.728824 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-podinfo" pod "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" (UID: "44d57d9c-ceaa-4136-ab40-82d9e89e6cc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:20:55.827015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.826969 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.827015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.827009 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.827015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.827018 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.907646 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.907604 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz"] Oct 30 01:20:55.908493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908459 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ece50684-1c45-48f6-b3a7-23f2aa6e888d" containerName="revoke-keys" Oct 30 01:20:55.908493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908487 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece50684-1c45-48f6-b3a7-23f2aa6e888d" containerName="revoke-keys" Oct 30 01:20:55.908723 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908505 4517 cpu_manager.go:414] "containerMap: removing container" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" Oct 30 01:20:55.908723 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908513 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" Oct 30 01:20:55.908723 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908559 4517 cpu_manager.go:414] "containerMap: removing container" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" Oct 30 01:20:55.908723 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908567 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" Oct 30 01:20:55.908723 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908576 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d17b1bc9-1b6b-499b-857c-994bb6283d0a" containerName="revoke-keys" Oct 30 01:20:55.908723 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908581 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17b1bc9-1b6b-499b-857c-994bb6283d0a" containerName="revoke-keys" Oct 30 01:20:55.908952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908753 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece50684-1c45-48f6-b3a7-23f2aa6e888d" containerName="revoke-keys" Oct 30 01:20:55.908952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908769 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" Oct 30 01:20:55.908952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908781 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerName="istio-proxy" Oct 30 01:20:55.908952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.908786 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17b1bc9-1b6b-499b-857c-994bb6283d0a" containerName="revoke-keys" Oct 30 01:20:55.914512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.914479 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:55.920210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.920161 4517 generic.go:356] "Generic (PLEG): container finished" podID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" containerID="c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f" exitCode=0 Oct 30 01:20:55.920431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.920218 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" Oct 30 01:20:55.920431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.920237 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" event={"ID":"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2","Type":"ContainerDied","Data":"c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f"} Oct 30 01:20:55.920431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.920313 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8" event={"ID":"44d57d9c-ceaa-4136-ab40-82d9e89e6cc2","Type":"ContainerDied","Data":"58e8919c4517a43304e32e524a3cc7fceeb02b3f81e07d7fe123c58b53d38959"} Oct 30 01:20:55.920431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.920347 4517 scope.go:122] "RemoveContainer" containerID="c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f" Oct 30 01:20:55.931171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.928405 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72gp\" (UniqueName: \"kubernetes.io/projected/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-kube-api-access-n72gp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:55.931171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.928461 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:55.931171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.928547 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:55.940066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.940024 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz"] Oct 30 01:20:55.950720 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.950672 4517 scope.go:122] "RemoveContainer" containerID="f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0" Oct 30 01:20:55.971053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.971021 4517 scope.go:122] "RemoveContainer" containerID="c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f" Oct 30 01:20:55.971560 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:20:55.971525 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f\": container with ID starting with c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f not found: ID does not exist" containerID="c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f" Oct 30 01:20:55.971638 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.971576 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f"} err="failed to get container status \"c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f\": rpc error: code = NotFound desc = could not find container \"c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f\": container with ID starting with c79355bc6dba8c92ebd9d9ca9a6af5980511a91adec6b8dcad9becaa6ccf514f not found: ID does not exist" Oct 30 01:20:55.971638 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.971606 4517 scope.go:122] "RemoveContainer" containerID="f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0" Oct 30 01:20:55.972229 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:20:55.972193 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0\": container with ID starting with f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0 not found: ID does not exist" containerID="f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0" Oct 30 01:20:55.972367 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.972238 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0"} err="failed to get container status \"f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0\": rpc error: code = NotFound desc = could not find container \"f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0\": container with ID starting with f24f8a1c322418e15541a57d779d7477772cdd21332e8c90ac692895e9a449c0 not found: ID does not exist" Oct 30 01:20:55.991025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:55.990986 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8"] Oct 30 01:20:56.001048 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.000972 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a42e5-openshift-default-84779c8749-k8vc8"] Oct 30 01:20:56.015185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.015148 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:20:56.015416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.015201 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:20:56.029770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.029673 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72gp\" (UniqueName: \"kubernetes.io/projected/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-kube-api-access-n72gp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:56.029971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.029780 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:56.029971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.029854 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:56.030474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.030446 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:56.032697 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.032668 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:56.051846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.051808 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72gp\" (UniqueName: \"kubernetes.io/projected/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-kube-api-access-n72gp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:56.238423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.238336 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d57d9c-ceaa-4136-ab40-82d9e89e6cc2" path="/var/lib/kubelet/pods/44d57d9c-ceaa-4136-ab40-82d9e89e6cc2/volumes" Oct 30 01:20:56.243398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.243365 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:56.376887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.376798 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:56.566110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.566059 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz"] Oct 30 01:20:56.572349 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:20:56.572313 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c5b9a8_c7ca_4c97_921d_30d30936f03d.slice/crio-83d6cb9b66935e9a0c3807c6bc3a7f9f1f3a14de346bbf801cbb2782e80656f1 WatchSource:0}: Error finding container 83d6cb9b66935e9a0c3807c6bc3a7f9f1f3a14de346bbf801cbb2782e80656f1: Status 404 returned error can't find the container with id 83d6cb9b66935e9a0c3807c6bc3a7f9f1f3a14de346bbf801cbb2782e80656f1 Oct 30 01:20:56.932029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:56.931929 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" event={"ID":"f0c5b9a8-c7ca-4c97-921d-30d30936f03d","Type":"ContainerStarted","Data":"83d6cb9b66935e9a0c3807c6bc3a7f9f1f3a14de346bbf801cbb2782e80656f1"} Oct 30 01:20:57.941847 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:57.941806 4517 generic.go:356] "Generic (PLEG): container finished" podID="f0c5b9a8-c7ca-4c97-921d-30d30936f03d" containerID="8a9f3b4da850bf1f702dd205c2584cd01e0e790b6ef7f4dc55492dba0c5735ae" exitCode=0 Oct 30 01:20:57.942308 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:57.941896 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" event={"ID":"f0c5b9a8-c7ca-4c97-921d-30d30936f03d","Type":"ContainerDied","Data":"8a9f3b4da850bf1f702dd205c2584cd01e0e790b6ef7f4dc55492dba0c5735ae"} Oct 30 01:20:58.921005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:58.920966 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:59.190551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.190472 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:59.206863 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.206827 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-kube-api-access-n72gp\" (UniqueName: \"kubernetes.io/projected/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-kube-api-access-n72gp\") pod \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " Oct 30 01:20:59.207070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.206923 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-openshift-ca\") pod \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " Oct 30 01:20:59.207070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.207055 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-xks-ca\") pod \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\" (UID: \"f0c5b9a8-c7ca-4c97-921d-30d30936f03d\") " Oct 30 01:20:59.207353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.207326 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-openshift-ca" pod "f0c5b9a8-c7ca-4c97-921d-30d30936f03d" (UID: "f0c5b9a8-c7ca-4c97-921d-30d30936f03d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:59.207561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.207537 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:59.209942 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.209914 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-xks-ca" pod "f0c5b9a8-c7ca-4c97-921d-30d30936f03d" (UID: "f0c5b9a8-c7ca-4c97-921d-30d30936f03d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:59.210071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.209956 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-kube-api-access-n72gp" pod "f0c5b9a8-c7ca-4c97-921d-30d30936f03d" (UID: "f0c5b9a8-c7ca-4c97-921d-30d30936f03d"). InnerVolumeSpecName "kube-api-access-n72gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:59.308570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.308506 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:59.308570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.308562 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n72gp\" (UniqueName: \"kubernetes.io/projected/f0c5b9a8-c7ca-4c97-921d-30d30936f03d-kube-api-access-n72gp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:20:59.959416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.959377 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" Oct 30 01:20:59.959672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.959433 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz" event={"ID":"f0c5b9a8-c7ca-4c97-921d-30d30936f03d","Type":"ContainerDied","Data":"83d6cb9b66935e9a0c3807c6bc3a7f9f1f3a14de346bbf801cbb2782e80656f1"} Oct 30 01:20:59.959672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:20:59.959494 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83d6cb9b66935e9a0c3807c6bc3a7f9f1f3a14de346bbf801cbb2782e80656f1" Oct 30 01:21:00.595750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:00.595665 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:00.798302 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:21:00.798061 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:21:01.329614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:01.329580 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:01.449129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:01.449080 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:01.899509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:01.899465 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.042732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:02.042577 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.247804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:02.205970 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.982249 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:02.982198 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:05.243429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:05.243377 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:06.214397 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:06.214348 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:06.431282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:06.429552 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:07.103002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:07.097308 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:07.808733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:07.806965 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:07.924242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:07.924200 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:08.048743 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:08.047678 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:08.321716 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:08.321664 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:08.433404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:08.433370 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:08.580255 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:08.580159 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:08.627862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:08.627826 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt"] Oct 30 01:21:08.807905 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:08.807861 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:08.944441 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:08.942387 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:09.087774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:09.084469 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:09.973677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:09.973642 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:11.632051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:11.632012 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:13.428808 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:13.428761 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:13.629239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:13.629177 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" podUID="b46f7744-bbd1-4bdd-a247-88f0135a1285" containerName="payload-processing" containerID="cri-o://ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb" gracePeriod=25 Oct 30 01:21:13.951616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:13.951578 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:21:14.065113 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.065068 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46f7744-bbd1-4bdd-a247-88f0135a1285-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46f7744-bbd1-4bdd-a247-88f0135a1285-plugins-config-volume\") pod \"b46f7744-bbd1-4bdd-a247-88f0135a1285\" (UID: \"b46f7744-bbd1-4bdd-a247-88f0135a1285\") " Oct 30 01:21:14.065903 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.065870 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46f7744-bbd1-4bdd-a247-88f0135a1285-kube-api-access-dlqkp\" (UniqueName: \"kubernetes.io/projected/b46f7744-bbd1-4bdd-a247-88f0135a1285-kube-api-access-dlqkp\") pod \"b46f7744-bbd1-4bdd-a247-88f0135a1285\" (UID: \"b46f7744-bbd1-4bdd-a247-88f0135a1285\") " Oct 30 01:21:14.066062 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.065944 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46f7744-bbd1-4bdd-a247-88f0135a1285-plugins-config-volume" pod "b46f7744-bbd1-4bdd-a247-88f0135a1285" (UID: "b46f7744-bbd1-4bdd-a247-88f0135a1285"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:14.067146 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.066553 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46f7744-bbd1-4bdd-a247-88f0135a1285-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:14.075002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.074932 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46f7744-bbd1-4bdd-a247-88f0135a1285-kube-api-access-dlqkp" pod "b46f7744-bbd1-4bdd-a247-88f0135a1285" (UID: "b46f7744-bbd1-4bdd-a247-88f0135a1285"). InnerVolumeSpecName "kube-api-access-dlqkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:14.167658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.167617 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqkp\" (UniqueName: \"kubernetes.io/projected/b46f7744-bbd1-4bdd-a247-88f0135a1285-kube-api-access-dlqkp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:14.690890 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.690842 4517 generic.go:356] "Generic (PLEG): container finished" podID="b46f7744-bbd1-4bdd-a247-88f0135a1285" containerID="ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb" exitCode=0 Oct 30 01:21:14.691451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.690891 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" event={"ID":"b46f7744-bbd1-4bdd-a247-88f0135a1285","Type":"ContainerDied","Data":"ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb"} Oct 30 01:21:14.691451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.690915 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" Oct 30 01:21:14.691451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.690971 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt" event={"ID":"b46f7744-bbd1-4bdd-a247-88f0135a1285","Type":"ContainerDied","Data":"5e8f384ce091fba016264a1e915b2a62227b58bf3c79819aa541be7a0726cfc9"} Oct 30 01:21:14.691451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.691003 4517 scope.go:122] "RemoveContainer" containerID="ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb" Oct 30 01:21:14.707685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.707580 4517 scope.go:122] "RemoveContainer" containerID="ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb" Oct 30 01:21:14.708076 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:21:14.708047 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb\": container with ID starting with ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb not found: ID does not exist" containerID="ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb" Oct 30 01:21:14.708172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.708086 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb"} err="failed to get container status \"ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb\": rpc error: code = NotFound desc = could not find container \"ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb\": container with ID starting with ba05d9c31f45c54ea8ac037dcc1170f7d1f0d0ef9c93f0b81bc0587306802abb not found: ID does not exist" Oct 30 01:21:14.742797 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.742676 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt"] Oct 30 01:21:14.752774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:14.752732 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-55fbd4b6cd-gtknt"] Oct 30 01:21:15.011275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.011159 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:15.134285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.134253 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:15.428342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.428159 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:15.895041 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.894993 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr"] Oct 30 01:21:15.895699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.895683 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f0c5b9a8-c7ca-4c97-921d-30d30936f03d" containerName="revoke-keys" Oct 30 01:21:15.895699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.895701 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c5b9a8-c7ca-4c97-921d-30d30936f03d" containerName="revoke-keys" Oct 30 01:21:15.895699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.895739 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b46f7744-bbd1-4bdd-a247-88f0135a1285" containerName="payload-processing" Oct 30 01:21:15.895699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.895744 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46f7744-bbd1-4bdd-a247-88f0135a1285" containerName="payload-processing" Oct 30 01:21:15.895978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.895892 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c5b9a8-c7ca-4c97-921d-30d30936f03d" containerName="revoke-keys" Oct 30 01:21:15.895978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.895902 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46f7744-bbd1-4bdd-a247-88f0135a1285" containerName="payload-processing" Oct 30 01:21:15.899504 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.899483 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:15.918416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.918384 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr"] Oct 30 01:21:15.975175 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.975136 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:21:15.975363 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:15.975197 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:21:16.017143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.017047 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxwj\" (UniqueName: \"kubernetes.io/projected/e5b867f4-9150-4165-b3f1-6981161a32d6-kube-api-access-mfxwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.017143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.017124 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.017391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.017205 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.118769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.118662 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxwj\" (UniqueName: \"kubernetes.io/projected/e5b867f4-9150-4165-b3f1-6981161a32d6-kube-api-access-mfxwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.118971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.118819 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.118971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.118845 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.119626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.119604 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.121569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.121534 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.140429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.140394 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxwj\" (UniqueName: \"kubernetes.io/projected/e5b867f4-9150-4165-b3f1-6981161a32d6-kube-api-access-mfxwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.214658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.214625 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:16.238337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.238297 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46f7744-bbd1-4bdd-a247-88f0135a1285" path="/var/lib/kubelet/pods/b46f7744-bbd1-4bdd-a247-88f0135a1285/volumes" Oct 30 01:21:16.546164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.546038 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr"] Oct 30 01:21:16.554402 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:21:16.554349 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b867f4_9150_4165_b3f1_6981161a32d6.slice/crio-7d3db8733df3cf7cb6cf9d832a92e90835780ccb5d3a7e6ffb0074a38e4abc8d WatchSource:0}: Error finding container 7d3db8733df3cf7cb6cf9d832a92e90835780ccb5d3a7e6ffb0074a38e4abc8d: Status 404 returned error can't find the container with id 7d3db8733df3cf7cb6cf9d832a92e90835780ccb5d3a7e6ffb0074a38e4abc8d Oct 30 01:21:16.748391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.748354 4517 generic.go:356] "Generic (PLEG): container finished" podID="35cf5570-1952-4038-af67-dded46578a90" containerID="eb2158fbbbb8d4b5224b7904584a5a9e03eb1c35239ed3e6f4aad2aa1111ff4a" exitCode=0 Oct 30 01:21:16.748597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.748429 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" event={"ID":"35cf5570-1952-4038-af67-dded46578a90","Type":"ContainerDied","Data":"eb2158fbbbb8d4b5224b7904584a5a9e03eb1c35239ed3e6f4aad2aa1111ff4a"} Oct 30 01:21:16.748907 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.748892 4517 scope.go:122] "RemoveContainer" containerID="eb2158fbbbb8d4b5224b7904584a5a9e03eb1c35239ed3e6f4aad2aa1111ff4a" Oct 30 01:21:16.750273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.750249 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" event={"ID":"e5b867f4-9150-4165-b3f1-6981161a32d6","Type":"ContainerStarted","Data":"7d3db8733df3cf7cb6cf9d832a92e90835780ccb5d3a7e6ffb0074a38e4abc8d"} Oct 30 01:21:16.751013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.750982 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:21:16.751082 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.751069 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:21:16.751125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:16.751105 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:21:17.051769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:17.051731 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:17.793860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:17.783074 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" event={"ID":"35cf5570-1952-4038-af67-dded46578a90","Type":"ContainerStarted","Data":"2663e73863508e507556d1ccadf38601f3a46c545ca15add3e9a4b3220d715f1"} Oct 30 01:21:17.793860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:17.790956 4517 generic.go:356] "Generic (PLEG): container finished" podID="e5b867f4-9150-4165-b3f1-6981161a32d6" containerID="4dda7b4b9a3db62cb0765f9174979f9e20e5088a3a36d39122e63c52bc5c9887" exitCode=0 Oct 30 01:21:17.793860 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:17.791116 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" event={"ID":"e5b867f4-9150-4165-b3f1-6981161a32d6","Type":"ContainerDied","Data":"4dda7b4b9a3db62cb0765f9174979f9e20e5088a3a36d39122e63c52bc5c9887"} Oct 30 01:21:18.438447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:18.438403 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:21:18.439660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:18.439633 4517 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 01:21:18.439829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:18.439683 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 01:21:18.911055 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:18.911008 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:19.044197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.044171 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:19.216130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.216033 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-openshift-ca\") pod \"e5b867f4-9150-4165-b3f1-6981161a32d6\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " Oct 30 01:21:19.216130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.216081 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b867f4-9150-4165-b3f1-6981161a32d6-kube-api-access-mfxwj\" (UniqueName: \"kubernetes.io/projected/e5b867f4-9150-4165-b3f1-6981161a32d6-kube-api-access-mfxwj\") pod \"e5b867f4-9150-4165-b3f1-6981161a32d6\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " Oct 30 01:21:19.216433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.216306 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-xks-ca\") pod \"e5b867f4-9150-4165-b3f1-6981161a32d6\" (UID: \"e5b867f4-9150-4165-b3f1-6981161a32d6\") " Oct 30 01:21:19.216498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.216460 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-openshift-ca" pod "e5b867f4-9150-4165-b3f1-6981161a32d6" (UID: "e5b867f4-9150-4165-b3f1-6981161a32d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:19.216983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.216959 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:19.219205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.219171 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b867f4-9150-4165-b3f1-6981161a32d6-kube-api-access-mfxwj" pod "e5b867f4-9150-4165-b3f1-6981161a32d6" (UID: "e5b867f4-9150-4165-b3f1-6981161a32d6"). InnerVolumeSpecName "kube-api-access-mfxwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:19.219314 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.219262 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-xks-ca" pod "e5b867f4-9150-4165-b3f1-6981161a32d6" (UID: "e5b867f4-9150-4165-b3f1-6981161a32d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:19.317885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.317845 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b867f4-9150-4165-b3f1-6981161a32d6-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:19.317885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.317874 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfxwj\" (UniqueName: \"kubernetes.io/projected/e5b867f4-9150-4165-b3f1-6981161a32d6-kube-api-access-mfxwj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:19.439518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.439483 4517 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 01:21:19.439991 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.439545 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 01:21:19.781305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.781269 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:19.891983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.891934 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" event={"ID":"e5b867f4-9150-4165-b3f1-6981161a32d6","Type":"ContainerDied","Data":"7d3db8733df3cf7cb6cf9d832a92e90835780ccb5d3a7e6ffb0074a38e4abc8d"} Oct 30 01:21:19.892354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.892335 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d3db8733df3cf7cb6cf9d832a92e90835780ccb5d3a7e6ffb0074a38e4abc8d" Oct 30 01:21:19.892580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.892565 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr" Oct 30 01:21:19.928766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:19.928699 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:21:20.439529 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:20.439492 4517 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 01:21:20.440038 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:20.439554 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 01:21:21.441740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:21.439223 4517 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 01:21:21.441740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:21.439296 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 01:21:21.608787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:21.608736 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:22.439451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:22.439407 4517 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 01:21:22.439762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:22.439483 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 01:21:23.031051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:23.031004 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw"] Oct 30 01:21:23.041113 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:23.041075 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b98f505-v5kgw"] Oct 30 01:21:23.439163 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:23.439126 4517 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 01:21:23.439429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:23.439398 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 01:21:23.478272 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:23.478232 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.641915 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:23.641874 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.763071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:23.762956 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:24.236939 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:24.236901 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92868a74-fbed-46a8-87c5-fc85ae88c7dd" path="/var/lib/kubelet/pods/92868a74-fbed-46a8-87c5-fc85ae88c7dd/volumes" Oct 30 01:21:24.444631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:24.444597 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:21:24.446091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:24.446071 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:21:25.398546 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:25.398499 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:26.310108 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:26.310070 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx"] Oct 30 01:21:26.342933 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:26.342892 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj"] Oct 30 01:21:26.452382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:26.452349 4517 scope.go:122] "RemoveContainer" containerID="a8e32d5ee5c0495959af19e2f0b3a6025736d180e9510b3dd2541c0862903ed8" Oct 30 01:21:26.457239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:26.457203 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:26.469859 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:26.469827 4517 scope.go:122] "RemoveContainer" containerID="eeadfe4d3331ccf76f0fa5142cb95fe73361adfcfbbe7643f692e15601d9a1b3" Oct 30 01:21:26.580006 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:26.579246 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:26.704994 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:26.704946 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:26.824440 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:26.824402 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:27.064933 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.064902 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt"] Oct 30 01:21:27.066506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.066478 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:27.102685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.102646 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w"] Oct 30 01:21:27.211332 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.211299 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:27.345793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.344052 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:27.905734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.904929 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm"] Oct 30 01:21:27.905734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.905637 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e5b867f4-9150-4165-b3f1-6981161a32d6" containerName="revoke-keys" Oct 30 01:21:27.905734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.905650 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b867f4-9150-4165-b3f1-6981161a32d6" containerName="revoke-keys" Oct 30 01:21:27.906391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.905851 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b867f4-9150-4165-b3f1-6981161a32d6" containerName="revoke-keys" Oct 30 01:21:27.913466 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.912748 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:27.927242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.926866 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp"] Oct 30 01:21:27.932547 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.928461 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:21:27.932547 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.928486 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:21:27.937657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.933151 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm"] Oct 30 01:21:27.947228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:27.947193 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4"] Oct 30 01:21:28.047801 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.047698 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.048174 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.048147 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.048355 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.048336 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d26vr\" (UniqueName: \"kubernetes.io/projected/920d52a7-a474-44e2-b737-e6ff1412df15-kube-api-access-d26vr\") pod \"maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.138631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.138524 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:28.150109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.150070 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.150330 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.150140 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d26vr\" (UniqueName: \"kubernetes.io/projected/920d52a7-a474-44e2-b737-e6ff1412df15-kube-api-access-d26vr\") pod \"maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.150499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.150473 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.153127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.153094 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.155881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.155816 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.173072 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.173038 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d26vr\" (UniqueName: \"kubernetes.io/projected/920d52a7-a474-44e2-b737-e6ff1412df15-kube-api-access-d26vr\") pod \"maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.250252 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.248838 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:28.329782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.327857 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:28.755612 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:21:28.755558 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920d52a7_a474_44e2_b737_e6ff1412df15.slice/crio-b76a3cb343eeafe66ac25ff7f09e78fca7e08146525aa527ab0e4d42f25d62d7 WatchSource:0}: Error finding container b76a3cb343eeafe66ac25ff7f09e78fca7e08146525aa527ab0e4d42f25d62d7: Status 404 returned error can't find the container with id b76a3cb343eeafe66ac25ff7f09e78fca7e08146525aa527ab0e4d42f25d62d7 Oct 30 01:21:28.768701 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:28.768475 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm"] Oct 30 01:21:29.059689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:29.059645 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" event={"ID":"920d52a7-a474-44e2-b737-e6ff1412df15","Type":"ContainerStarted","Data":"b76a3cb343eeafe66ac25ff7f09e78fca7e08146525aa527ab0e4d42f25d62d7"} Oct 30 01:21:29.402233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:29.402133 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:29.868210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:29.865235 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:30.099806 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:30.098320 4517 generic.go:356] "Generic (PLEG): container finished" podID="920d52a7-a474-44e2-b737-e6ff1412df15" containerID="1ec71b374d134ac6dac0cd7db10fedab435981075240a80f143bc926bb582922" exitCode=0 Oct 30 01:21:30.099806 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:30.098439 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" event={"ID":"920d52a7-a474-44e2-b737-e6ff1412df15","Type":"ContainerDied","Data":"1ec71b374d134ac6dac0cd7db10fedab435981075240a80f143bc926bb582922"} Oct 30 01:21:30.762922 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:21:30.762867 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:21:31.292687 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.292657 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:31.310556 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.310516 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" podUID="1885f34e-6420-4b73-aab4-18d173a0f95a" containerName="payload-processing" containerID="cri-o://6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c" gracePeriod=25 Oct 30 01:21:31.344294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.344236 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" podUID="25333518-39ac-4e6c-a406-80a09d216b14" containerName="payload-pre-processing" containerID="cri-o://5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767" gracePeriod=25 Oct 30 01:21:31.353249 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.353210 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:31.421790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.411434 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920d52a7-a474-44e2-b737-e6ff1412df15-kube-api-access-d26vr\" (UniqueName: \"kubernetes.io/projected/920d52a7-a474-44e2-b737-e6ff1412df15-kube-api-access-d26vr\") pod \"920d52a7-a474-44e2-b737-e6ff1412df15\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " Oct 30 01:21:31.421790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.412950 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-xks-ca\") pod \"920d52a7-a474-44e2-b737-e6ff1412df15\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " Oct 30 01:21:31.421790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.413575 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-openshift-ca" pod "920d52a7-a474-44e2-b737-e6ff1412df15" (UID: "920d52a7-a474-44e2-b737-e6ff1412df15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:31.421790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.413731 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-openshift-ca\") pod \"920d52a7-a474-44e2-b737-e6ff1412df15\" (UID: \"920d52a7-a474-44e2-b737-e6ff1412df15\") " Oct 30 01:21:31.452043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.442086 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:31.471940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.470089 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-xks-ca" pod "920d52a7-a474-44e2-b737-e6ff1412df15" (UID: "920d52a7-a474-44e2-b737-e6ff1412df15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:31.471940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.470377 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920d52a7-a474-44e2-b737-e6ff1412df15-kube-api-access-d26vr" pod "920d52a7-a474-44e2-b737-e6ff1412df15" (UID: "920d52a7-a474-44e2-b737-e6ff1412df15"). InnerVolumeSpecName "kube-api-access-d26vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:31.544512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.544470 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d26vr\" (UniqueName: \"kubernetes.io/projected/920d52a7-a474-44e2-b737-e6ff1412df15-kube-api-access-d26vr\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:31.544512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:31.544514 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920d52a7-a474-44e2-b737-e6ff1412df15-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.066188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.066127 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" podUID="c485ddbf-39d0-4c2f-b45c-4900a9775566" containerName="payload-processing" containerID="cri-o://33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac" gracePeriod=25 Oct 30 01:21:32.106492 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.106441 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" podUID="ba845c42-6120-4c91-b619-5d03dd41393e" containerName="payload-pre-processing" containerID="cri-o://5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b" gracePeriod=25 Oct 30 01:21:32.121304 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.121270 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" Oct 30 01:21:32.121304 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.121280 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm" event={"ID":"920d52a7-a474-44e2-b737-e6ff1412df15","Type":"ContainerDied","Data":"b76a3cb343eeafe66ac25ff7f09e78fca7e08146525aa527ab0e4d42f25d62d7"} Oct 30 01:21:32.121514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.121333 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b76a3cb343eeafe66ac25ff7f09e78fca7e08146525aa527ab0e4d42f25d62d7" Oct 30 01:21:32.425526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.425477 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:21:32.535906 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.535856 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:21:32.558455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.558422 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c485ddbf-39d0-4c2f-b45c-4900a9775566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c485ddbf-39d0-4c2f-b45c-4900a9775566-plugins-config-volume\") pod \"c485ddbf-39d0-4c2f-b45c-4900a9775566\" (UID: \"c485ddbf-39d0-4c2f-b45c-4900a9775566\") " Oct 30 01:21:32.558677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.558486 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c485ddbf-39d0-4c2f-b45c-4900a9775566-kube-api-access-b5kjq\" (UniqueName: \"kubernetes.io/projected/c485ddbf-39d0-4c2f-b45c-4900a9775566-kube-api-access-b5kjq\") pod \"c485ddbf-39d0-4c2f-b45c-4900a9775566\" (UID: \"c485ddbf-39d0-4c2f-b45c-4900a9775566\") " Oct 30 01:21:32.558883 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.558845 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c485ddbf-39d0-4c2f-b45c-4900a9775566-plugins-config-volume" pod "c485ddbf-39d0-4c2f-b45c-4900a9775566" (UID: "c485ddbf-39d0-4c2f-b45c-4900a9775566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:32.560080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.560050 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c485ddbf-39d0-4c2f-b45c-4900a9775566-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.561269 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.561244 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c485ddbf-39d0-4c2f-b45c-4900a9775566-kube-api-access-b5kjq" pod "c485ddbf-39d0-4c2f-b45c-4900a9775566" (UID: "c485ddbf-39d0-4c2f-b45c-4900a9775566"). InnerVolumeSpecName "kube-api-access-b5kjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:32.660747 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.660641 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba845c42-6120-4c91-b619-5d03dd41393e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba845c42-6120-4c91-b619-5d03dd41393e-plugins-config-volume\") pod \"ba845c42-6120-4c91-b619-5d03dd41393e\" (UID: \"ba845c42-6120-4c91-b619-5d03dd41393e\") " Oct 30 01:21:32.660747 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.660695 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba845c42-6120-4c91-b619-5d03dd41393e-kube-api-access-9kwjv\" (UniqueName: \"kubernetes.io/projected/ba845c42-6120-4c91-b619-5d03dd41393e-kube-api-access-9kwjv\") pod \"ba845c42-6120-4c91-b619-5d03dd41393e\" (UID: \"ba845c42-6120-4c91-b619-5d03dd41393e\") " Oct 30 01:21:32.661188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.661155 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba845c42-6120-4c91-b619-5d03dd41393e-plugins-config-volume" pod "ba845c42-6120-4c91-b619-5d03dd41393e" (UID: "ba845c42-6120-4c91-b619-5d03dd41393e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:32.661338 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.661228 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5kjq\" (UniqueName: \"kubernetes.io/projected/c485ddbf-39d0-4c2f-b45c-4900a9775566-kube-api-access-b5kjq\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.663177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.663143 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba845c42-6120-4c91-b619-5d03dd41393e-kube-api-access-9kwjv" pod "ba845c42-6120-4c91-b619-5d03dd41393e" (UID: "ba845c42-6120-4c91-b619-5d03dd41393e"). InnerVolumeSpecName "kube-api-access-9kwjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:32.762348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.762307 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba845c42-6120-4c91-b619-5d03dd41393e-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.762348 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.762342 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kwjv\" (UniqueName: \"kubernetes.io/projected/ba845c42-6120-4c91-b619-5d03dd41393e-kube-api-access-9kwjv\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.927510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.927385 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" podUID="1bb00a4d-eb99-4bc2-a152-ec99247ac45c" containerName="payload-processing" containerID="cri-o://ae9f3dcce395dd6b6a1b86391fd771c249b7e400148426ad3eb75038d984d8ee" gracePeriod=25 Oct 30 01:21:32.947826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:32.947773 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" podUID="01bd6e21-5e90-4063-956d-09f0b7523bd2" containerName="payload-pre-processing" containerID="cri-o://9baff3c621438c91f460a8e6bfdebb1d2ebfd060781e2d5ce1e0338b52c51b73" gracePeriod=25 Oct 30 01:21:33.032448 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.032390 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k"] Oct 30 01:21:33.036746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.036692 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61ad374b-w2n5k"] Oct 30 01:21:33.124679 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.124643 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:33.134798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.134536 4517 generic.go:356] "Generic (PLEG): container finished" podID="01bd6e21-5e90-4063-956d-09f0b7523bd2" containerID="9baff3c621438c91f460a8e6bfdebb1d2ebfd060781e2d5ce1e0338b52c51b73" exitCode=0 Oct 30 01:21:33.134798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.134685 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" event={"ID":"01bd6e21-5e90-4063-956d-09f0b7523bd2","Type":"ContainerDied","Data":"9baff3c621438c91f460a8e6bfdebb1d2ebfd060781e2d5ce1e0338b52c51b73"} Oct 30 01:21:33.137347 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.137314 4517 generic.go:356] "Generic (PLEG): container finished" podID="1bb00a4d-eb99-4bc2-a152-ec99247ac45c" containerID="ae9f3dcce395dd6b6a1b86391fd771c249b7e400148426ad3eb75038d984d8ee" exitCode=0 Oct 30 01:21:33.137921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.137858 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" event={"ID":"1bb00a4d-eb99-4bc2-a152-ec99247ac45c","Type":"ContainerDied","Data":"ae9f3dcce395dd6b6a1b86391fd771c249b7e400148426ad3eb75038d984d8ee"} Oct 30 01:21:33.140270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.140233 4517 generic.go:356] "Generic (PLEG): container finished" podID="ba845c42-6120-4c91-b619-5d03dd41393e" containerID="5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b" exitCode=0 Oct 30 01:21:33.140514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.140491 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" event={"ID":"ba845c42-6120-4c91-b619-5d03dd41393e","Type":"ContainerDied","Data":"5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b"} Oct 30 01:21:33.140594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.140554 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" event={"ID":"ba845c42-6120-4c91-b619-5d03dd41393e","Type":"ContainerDied","Data":"a03c811bf821dd8c74e8410f4ba4fef878a6267005ac41e0cb9bfc425a08be09"} Oct 30 01:21:33.140594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.140585 4517 scope.go:122] "RemoveContainer" containerID="5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b" Oct 30 01:21:33.140785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.140770 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w" Oct 30 01:21:33.149792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.149632 4517 generic.go:356] "Generic (PLEG): container finished" podID="c485ddbf-39d0-4c2f-b45c-4900a9775566" containerID="33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac" exitCode=0 Oct 30 01:21:33.149792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.149693 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" event={"ID":"c485ddbf-39d0-4c2f-b45c-4900a9775566","Type":"ContainerDied","Data":"33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac"} Oct 30 01:21:33.152851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.150159 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" event={"ID":"c485ddbf-39d0-4c2f-b45c-4900a9775566","Type":"ContainerDied","Data":"155ccadb9d8ba7f09215e2553be009e07305105acd14b14f939cd6f85a9f3ce8"} Oct 30 01:21:33.152851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.150319 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt" Oct 30 01:21:33.293001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.292958 4517 scope.go:122] "RemoveContainer" containerID="5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b" Oct 30 01:21:33.293768 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:21:33.293737 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b\": container with ID starting with 5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b not found: ID does not exist" containerID="5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b" Oct 30 01:21:33.295024 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.294957 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b"} err="failed to get container status \"5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b\": rpc error: code = NotFound desc = could not find container \"5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b\": container with ID starting with 5476d0193d9f1926582dff4eb56f809609f559260f76511d55f49bf52d919a0b not found: ID does not exist" Oct 30 01:21:33.295223 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.295198 4517 scope.go:122] "RemoveContainer" containerID="33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac" Oct 30 01:21:33.297932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.297894 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w"] Oct 30 01:21:33.304356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.304166 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-76d444555-7qh5w"] Oct 30 01:21:33.335098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.335053 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt"] Oct 30 01:21:33.342871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.342835 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-65ffff6bdd-wskgt"] Oct 30 01:21:33.356482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.356432 4517 scope.go:122] "RemoveContainer" containerID="33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac" Oct 30 01:21:33.357382 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:21:33.357344 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac\": container with ID starting with 33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac not found: ID does not exist" containerID="33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac" Oct 30 01:21:33.357513 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.357391 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac"} err="failed to get container status \"33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac\": rpc error: code = NotFound desc = could not find container \"33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac\": container with ID starting with 33f91c63f40d4bc37744b8948020e62d415522ea714c4febc225b5e7bb7ba3ac not found: ID does not exist" Oct 30 01:21:33.361924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.361899 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:21:33.409431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.409404 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-kube-api-access-xppcq\" (UniqueName: \"kubernetes.io/projected/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-kube-api-access-xppcq\") pod \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\" (UID: \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\") " Oct 30 01:21:33.409622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.409480 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-plugins-config-volume\") pod \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\" (UID: \"1bb00a4d-eb99-4bc2-a152-ec99247ac45c\") " Oct 30 01:21:33.410599 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.410559 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-plugins-config-volume" pod "1bb00a4d-eb99-4bc2-a152-ec99247ac45c" (UID: "1bb00a4d-eb99-4bc2-a152-ec99247ac45c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:33.414952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.414917 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-kube-api-access-xppcq" pod "1bb00a4d-eb99-4bc2-a152-ec99247ac45c" (UID: "1bb00a4d-eb99-4bc2-a152-ec99247ac45c"). InnerVolumeSpecName "kube-api-access-xppcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:33.421783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.421751 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:21:33.510881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.510774 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01bd6e21-5e90-4063-956d-09f0b7523bd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bd6e21-5e90-4063-956d-09f0b7523bd2-plugins-config-volume\") pod \"01bd6e21-5e90-4063-956d-09f0b7523bd2\" (UID: \"01bd6e21-5e90-4063-956d-09f0b7523bd2\") " Oct 30 01:21:33.510881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.510826 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01bd6e21-5e90-4063-956d-09f0b7523bd2-kube-api-access-ptt2v\" (UniqueName: \"kubernetes.io/projected/01bd6e21-5e90-4063-956d-09f0b7523bd2-kube-api-access-ptt2v\") pod \"01bd6e21-5e90-4063-956d-09f0b7523bd2\" (UID: \"01bd6e21-5e90-4063-956d-09f0b7523bd2\") " Oct 30 01:21:33.511478 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.511361 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01bd6e21-5e90-4063-956d-09f0b7523bd2-plugins-config-volume" pod "01bd6e21-5e90-4063-956d-09f0b7523bd2" (UID: "01bd6e21-5e90-4063-956d-09f0b7523bd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:33.511603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.511571 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xppcq\" (UniqueName: \"kubernetes.io/projected/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-kube-api-access-xppcq\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:33.511603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.511602 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb00a4d-eb99-4bc2-a152-ec99247ac45c-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:33.511836 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.511617 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bd6e21-5e90-4063-956d-09f0b7523bd2-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:33.513813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.513779 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01bd6e21-5e90-4063-956d-09f0b7523bd2-kube-api-access-ptt2v" pod "01bd6e21-5e90-4063-956d-09f0b7523bd2" (UID: "01bd6e21-5e90-4063-956d-09f0b7523bd2"). InnerVolumeSpecName "kube-api-access-ptt2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:33.612506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.612473 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptt2v\" (UniqueName: \"kubernetes.io/projected/01bd6e21-5e90-4063-956d-09f0b7523bd2-kube-api-access-ptt2v\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:33.706131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.706094 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" podUID="1885f34e-6420-4b73-aab4-18d173a0f95a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:33.706334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:33.706142 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" podUID="25333518-39ac-4e6c-a406-80a09d216b14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:34.159759 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.159729 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" Oct 30 01:21:34.159957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.159732 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4" event={"ID":"01bd6e21-5e90-4063-956d-09f0b7523bd2","Type":"ContainerDied","Data":"cbfc1e0211482a9ee6b790d6e9a8ece46d26d8c8a1c78d0ceac51c22997d7434"} Oct 30 01:21:34.159957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.159890 4517 scope.go:122] "RemoveContainer" containerID="9baff3c621438c91f460a8e6bfdebb1d2ebfd060781e2d5ce1e0338b52c51b73" Oct 30 01:21:34.161352 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.161325 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" event={"ID":"1bb00a4d-eb99-4bc2-a152-ec99247ac45c","Type":"ContainerDied","Data":"d1500de065adb9591f6c4e588bb9767a1fb52d449a235c8b186a993cf9e5e722"} Oct 30 01:21:34.161527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.161336 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp" Oct 30 01:21:34.174444 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.174415 4517 scope.go:122] "RemoveContainer" containerID="ae9f3dcce395dd6b6a1b86391fd771c249b7e400148426ad3eb75038d984d8ee" Oct 30 01:21:34.236663 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.236625 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7adb12b2-39c9-474d-aafa-621b23281665" path="/var/lib/kubelet/pods/7adb12b2-39c9-474d-aafa-621b23281665/volumes" Oct 30 01:21:34.237166 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.237117 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba845c42-6120-4c91-b619-5d03dd41393e" path="/var/lib/kubelet/pods/ba845c42-6120-4c91-b619-5d03dd41393e/volumes" Oct 30 01:21:34.237606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.237583 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c485ddbf-39d0-4c2f-b45c-4900a9775566" path="/var/lib/kubelet/pods/c485ddbf-39d0-4c2f-b45c-4900a9775566/volumes" Oct 30 01:21:34.270469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.270436 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp"] Oct 30 01:21:34.300429 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.300388 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-d6577bb7b-zwdzp"] Oct 30 01:21:34.334046 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.333999 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4"] Oct 30 01:21:34.338139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.338103 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-5978954f55-fgsx4"] Oct 30 01:21:34.834011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:34.833976 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:36.183958 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:36.183847 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:36.241351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:36.241297 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01bd6e21-5e90-4063-956d-09f0b7523bd2" path="/var/lib/kubelet/pods/01bd6e21-5e90-4063-956d-09f0b7523bd2/volumes" Oct 30 01:21:36.242054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:36.242026 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb00a4d-eb99-4bc2-a152-ec99247ac45c" path="/var/lib/kubelet/pods/1bb00a4d-eb99-4bc2-a152-ec99247ac45c/volumes" Oct 30 01:21:36.833847 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:36.833779 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:36.944858 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:36.944825 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:36.960276 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:36.960243 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85"] Oct 30 01:21:36.970177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:36.970145 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b3593a-78c85"] Oct 30 01:21:37.064636 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:37.064600 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:37.455510 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:37.455468 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:38.236870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:38.236831 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115ac86b-e232-4ac4-93ce-72e8a0d20541" path="/var/lib/kubelet/pods/115ac86b-e232-4ac4-93ce-72e8a0d20541/volumes" Oct 30 01:21:39.434282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:39.434248 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:41.049121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:41.049081 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:42.617015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:42.616980 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:43.398878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:43.398830 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:43.520996 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:43.520959 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:43.683802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:43.683643 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r"] Oct 30 01:21:43.684672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:43.684615 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" containerID="cri-o://b02c16a9c48ba52328b0103ed68be0c3a47e1467925c27a3180bbe89768882d2" gracePeriod=30 Oct 30 01:21:43.708544 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:43.708078 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" podUID="25333518-39ac-4e6c-a406-80a09d216b14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:43.737072 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:43.737034 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:43.740885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:43.739831 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" podUID="1885f34e-6420-4b73-aab4-18d173a0f95a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:44.163601 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:44.163385 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:44.323232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:44.323177 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:44.480965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:44.480851 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:44.846052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:44.846016 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:44.968438 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:44.968400 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:45.098865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:45.098301 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:45.506236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:45.506144 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:46.028497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:46.028460 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww"] Oct 30 01:21:46.038285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:46.038246 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a1de2e-cwqww"] Oct 30 01:21:46.236213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:46.236176 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ea793b-b7d2-4fad-8347-f73c4667467c" path="/var/lib/kubelet/pods/65ea793b-b7d2-4fad-8347-f73c4667467c/volumes" Oct 30 01:21:46.534123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:46.534026 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:46.695942 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:46.695891 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:46.815211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:46.815170 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:47.197598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:47.196931 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:47.406544 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:47.406507 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:47.432295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:47.432255 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm"] Oct 30 01:21:47.443965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:47.443930 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d1ae36-l2pvm"] Oct 30 01:21:47.588356 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:47.588317 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:47.944589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:47.944499 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:48.238572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:48.238486 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920d52a7-a474-44e2-b737-e6ff1412df15" path="/var/lib/kubelet/pods/920d52a7-a474-44e2-b737-e6ff1412df15/volumes" Oct 30 01:21:49.009883 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.004682 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:49.244808 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.235652 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:49.334002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.333963 4517 generic.go:356] "Generic (PLEG): container finished" podID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerID="b02c16a9c48ba52328b0103ed68be0c3a47e1467925c27a3180bbe89768882d2" exitCode=0 Oct 30 01:21:49.334391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.334364 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" event={"ID":"920fe4a0-48c7-4b43-a746-0543f8b3e9bf","Type":"ContainerDied","Data":"b02c16a9c48ba52328b0103ed68be0c3a47e1467925c27a3180bbe89768882d2"} Oct 30 01:21:49.738793 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.738760 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.791122 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-socket\") pod \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.791204 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-token\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-token\") pod \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.791289 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-data\") pod \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.791321 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-podinfo\") pod \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.791472 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-envoy\") pod \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.791600 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-kube-api-access-zmfvz\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-kube-api-access-zmfvz\") pod \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.791639 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-credential-socket\") pod \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.792213 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-data" pod "920fe4a0-48c7-4b43-a746-0543f8b3e9bf" (UID: "920fe4a0-48c7-4b43-a746-0543f8b3e9bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.792593 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-socket" pod "920fe4a0-48c7-4b43-a746-0543f8b3e9bf" (UID: "920fe4a0-48c7-4b43-a746-0543f8b3e9bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.793503 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-certs\") pod \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " Oct 30 01:21:49.793769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.793684 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istiod-ca-cert\") pod \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\" (UID: \"920fe4a0-48c7-4b43-a746-0543f8b3e9bf\") " Oct 30 01:21:49.794775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.794569 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.794775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.794596 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.795054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.795029 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istiod-ca-cert" pod "920fe4a0-48c7-4b43-a746-0543f8b3e9bf" (UID: "920fe4a0-48c7-4b43-a746-0543f8b3e9bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:49.799481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.799444 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-credential-socket" pod "920fe4a0-48c7-4b43-a746-0543f8b3e9bf" (UID: "920fe4a0-48c7-4b43-a746-0543f8b3e9bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.801935 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.801902 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-certs" pod "920fe4a0-48c7-4b43-a746-0543f8b3e9bf" (UID: "920fe4a0-48c7-4b43-a746-0543f8b3e9bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.811597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.811550 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-podinfo" pod "920fe4a0-48c7-4b43-a746-0543f8b3e9bf" (UID: "920fe4a0-48c7-4b43-a746-0543f8b3e9bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:49.821484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.821370 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-kube-api-access-zmfvz" pod "920fe4a0-48c7-4b43-a746-0543f8b3e9bf" (UID: "920fe4a0-48c7-4b43-a746-0543f8b3e9bf"). InnerVolumeSpecName "kube-api-access-zmfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:49.821484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.821448 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-token" pod "920fe4a0-48c7-4b43-a746-0543f8b3e9bf" (UID: "920fe4a0-48c7-4b43-a746-0543f8b3e9bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:49.827110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.827062 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-envoy" pod "920fe4a0-48c7-4b43-a746-0543f8b3e9bf" (UID: "920fe4a0-48c7-4b43-a746-0543f8b3e9bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.896924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.896687 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.896924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.896787 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.896924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.896803 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.896924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.896817 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.896924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.896832 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfvz\" (UniqueName: \"kubernetes.io/projected/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-kube-api-access-zmfvz\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.896924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.896850 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.896924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:49.896867 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920fe4a0-48c7-4b43-a746-0543f8b3e9bf-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:50.005639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.005484 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:50.408449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.401744 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" event={"ID":"920fe4a0-48c7-4b43-a746-0543f8b3e9bf","Type":"ContainerDied","Data":"d8d91d613c9aeb253a4b43e8aeb43391e6b03c828585ca9ed40d353b8b57b961"} Oct 30 01:21:50.408449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.401833 4517 scope.go:122] "RemoveContainer" containerID="b02c16a9c48ba52328b0103ed68be0c3a47e1467925c27a3180bbe89768882d2" Oct 30 01:21:50.408449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.402060 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r" Oct 30 01:21:50.615171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.615069 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r"] Oct 30 01:21:50.623022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.622953 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-q524r"] Oct 30 01:21:50.655506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.655279 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:50.732850 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.732803 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz"] Oct 30 01:21:50.739223 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.739186 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-33e759-xzxpz"] Oct 30 01:21:50.767957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:50.767911 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:51.220398 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.220366 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:51.665651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.665616 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 01:21:51.697526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.697500 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 01:21:51.815053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.814992 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1885f34e-6420-4b73-aab4-18d173a0f95a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1885f34e-6420-4b73-aab4-18d173a0f95a-plugins-config-volume\") pod \"1885f34e-6420-4b73-aab4-18d173a0f95a\" (UID: \"1885f34e-6420-4b73-aab4-18d173a0f95a\") " Oct 30 01:21:51.815284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.815167 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25333518-39ac-4e6c-a406-80a09d216b14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25333518-39ac-4e6c-a406-80a09d216b14-plugins-config-volume\") pod \"25333518-39ac-4e6c-a406-80a09d216b14\" (UID: \"25333518-39ac-4e6c-a406-80a09d216b14\") " Oct 30 01:21:51.815284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.815223 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25333518-39ac-4e6c-a406-80a09d216b14-kube-api-access-v8ksf\" (UniqueName: \"kubernetes.io/projected/25333518-39ac-4e6c-a406-80a09d216b14-kube-api-access-v8ksf\") pod \"25333518-39ac-4e6c-a406-80a09d216b14\" (UID: \"25333518-39ac-4e6c-a406-80a09d216b14\") " Oct 30 01:21:51.815447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.815412 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1885f34e-6420-4b73-aab4-18d173a0f95a-kube-api-access-fdqqg\" (UniqueName: \"kubernetes.io/projected/1885f34e-6420-4b73-aab4-18d173a0f95a-kube-api-access-fdqqg\") pod \"1885f34e-6420-4b73-aab4-18d173a0f95a\" (UID: \"1885f34e-6420-4b73-aab4-18d173a0f95a\") " Oct 30 01:21:51.815517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.815391 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1885f34e-6420-4b73-aab4-18d173a0f95a-plugins-config-volume" pod "1885f34e-6420-4b73-aab4-18d173a0f95a" (UID: "1885f34e-6420-4b73-aab4-18d173a0f95a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:51.815624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.815590 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25333518-39ac-4e6c-a406-80a09d216b14-plugins-config-volume" pod "25333518-39ac-4e6c-a406-80a09d216b14" (UID: "25333518-39ac-4e6c-a406-80a09d216b14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:51.816014 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.815992 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1885f34e-6420-4b73-aab4-18d173a0f95a-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:51.816139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.816016 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25333518-39ac-4e6c-a406-80a09d216b14-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:51.817774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.817740 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25333518-39ac-4e6c-a406-80a09d216b14-kube-api-access-v8ksf" pod "25333518-39ac-4e6c-a406-80a09d216b14" (UID: "25333518-39ac-4e6c-a406-80a09d216b14"). InnerVolumeSpecName "kube-api-access-v8ksf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:51.817871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.817779 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1885f34e-6420-4b73-aab4-18d173a0f95a-kube-api-access-fdqqg" pod "1885f34e-6420-4b73-aab4-18d173a0f95a" (UID: "1885f34e-6420-4b73-aab4-18d173a0f95a"). InnerVolumeSpecName "kube-api-access-fdqqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:51.917770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.917703 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdqqg\" (UniqueName: \"kubernetes.io/projected/1885f34e-6420-4b73-aab4-18d173a0f95a-kube-api-access-fdqqg\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:51.917770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:51.917766 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ksf\" (UniqueName: \"kubernetes.io/projected/25333518-39ac-4e6c-a406-80a09d216b14-kube-api-access-v8ksf\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.236725 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.236629 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" path="/var/lib/kubelet/pods/920fe4a0-48c7-4b43-a746-0543f8b3e9bf/volumes" Oct 30 01:21:52.237081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.237066 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c5b9a8-c7ca-4c97-921d-30d30936f03d" path="/var/lib/kubelet/pods/f0c5b9a8-c7ca-4c97-921d-30d30936f03d/volumes" Oct 30 01:21:52.400628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.400577 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:52.421572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.421542 4517 generic.go:356] "Generic (PLEG): container finished" podID="25333518-39ac-4e6c-a406-80a09d216b14" containerID="5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767" exitCode=0 Oct 30 01:21:52.421782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.421641 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" Oct 30 01:21:52.421782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.421680 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" event={"ID":"25333518-39ac-4e6c-a406-80a09d216b14","Type":"ContainerDied","Data":"5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767"} Oct 30 01:21:52.421782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.421744 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj" event={"ID":"25333518-39ac-4e6c-a406-80a09d216b14","Type":"ContainerDied","Data":"12ee10c97e4489286741ede947df9998faff9245b80865b0ddf5958f71dc0874"} Oct 30 01:21:52.421782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.421774 4517 scope.go:122] "RemoveContainer" containerID="5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767" Oct 30 01:21:52.423645 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.423624 4517 generic.go:356] "Generic (PLEG): container finished" podID="1885f34e-6420-4b73-aab4-18d173a0f95a" containerID="6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c" exitCode=0 Oct 30 01:21:52.423805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.423675 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" Oct 30 01:21:52.423805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.423674 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" event={"ID":"1885f34e-6420-4b73-aab4-18d173a0f95a","Type":"ContainerDied","Data":"6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c"} Oct 30 01:21:52.423805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.423768 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx" event={"ID":"1885f34e-6420-4b73-aab4-18d173a0f95a","Type":"ContainerDied","Data":"cdddbf9cd832d01d7e3d8fa6bb00ffa9f93638a1831f71d72134a003b9cfe5b4"} Oct 30 01:21:52.432966 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.432950 4517 scope.go:122] "RemoveContainer" containerID="5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767" Oct 30 01:21:52.433326 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:21:52.433300 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767\": container with ID starting with 5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767 not found: ID does not exist" containerID="5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767" Oct 30 01:21:52.433400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.433336 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767"} err="failed to get container status \"5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767\": rpc error: code = NotFound desc = could not find container \"5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767\": container with ID starting with 5a808d68cc167ada2d78be2f31990ac8f30589f0d879c44e8baee64189e40767 not found: ID does not exist" Oct 30 01:21:52.433400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.433352 4517 scope.go:122] "RemoveContainer" containerID="6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c" Oct 30 01:21:52.443600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.443580 4517 scope.go:122] "RemoveContainer" containerID="6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c" Oct 30 01:21:52.443984 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:21:52.443958 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c\": container with ID starting with 6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c not found: ID does not exist" containerID="6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c" Oct 30 01:21:52.444101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.443995 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c"} err="failed to get container status \"6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c\": rpc error: code = NotFound desc = could not find container \"6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c\": container with ID starting with 6c2bba642ea2c6f4a69090c79c01aec393af4cfa7f8d186be6b1fffa28b43c9c not found: ID does not exist" Oct 30 01:21:52.512115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.512010 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:52.512622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.512602 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx"] Oct 30 01:21:52.516244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.516211 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-62f5d1-6649996454-bqvnx"] Oct 30 01:21:52.543695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.543349 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj"] Oct 30 01:21:52.547149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.547117 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-62f5d1-5b9c4dc69f-hnzwj"] Oct 30 01:21:52.638231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.638159 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:52.772082 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:52.771923 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:54.237233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:54.237192 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1885f34e-6420-4b73-aab4-18d173a0f95a" path="/var/lib/kubelet/pods/1885f34e-6420-4b73-aab4-18d173a0f95a/volumes" Oct 30 01:21:54.237682 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:54.237666 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25333518-39ac-4e6c-a406-80a09d216b14" path="/var/lib/kubelet/pods/25333518-39ac-4e6c-a406-80a09d216b14/volumes" Oct 30 01:21:54.514805 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:54.513698 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:54.770502 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:54.770356 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:54.892404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:54.892367 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:55.042736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:55.042677 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk"] Oct 30 01:21:55.072766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:55.068501 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630db937-kcbtk"] Oct 30 01:21:55.132559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:55.132506 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:56.236959 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:56.236914 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3ce1c0-ca57-48b0-a758-08177f5f0af5" path="/var/lib/kubelet/pods/7c3ce1c0-ca57-48b0-a758-08177f5f0af5/volumes" Oct 30 01:21:56.405734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:56.404637 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:57.806214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:57.806180 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j"] Oct 30 01:21:57.806689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:57.806435 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" podUID="aa6bfebb-1bf7-4279-82c8-bd98ec90d134" containerName="maas-api" containerID="cri-o://f6ad25c0c492fc00ac2c0e617b8464c0ac0f0007a98e1f64b43b850a2e13c24e" gracePeriod=30 Oct 30 01:21:57.879822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:57.879612 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:58.523995 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.523906 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:58.548910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.548877 4517 generic.go:356] "Generic (PLEG): container finished" podID="aa6bfebb-1bf7-4279-82c8-bd98ec90d134" containerID="f6ad25c0c492fc00ac2c0e617b8464c0ac0f0007a98e1f64b43b850a2e13c24e" exitCode=0 Oct 30 01:21:58.549060 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.548952 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" event={"ID":"aa6bfebb-1bf7-4279-82c8-bd98ec90d134","Type":"ContainerDied","Data":"f6ad25c0c492fc00ac2c0e617b8464c0ac0f0007a98e1f64b43b850a2e13c24e"} Oct 30 01:21:58.569754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.569673 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr"] Oct 30 01:21:58.572897 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.572863 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 01:21:58.578692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.578657 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62f5d1-9wxtr"] Oct 30 01:21:58.634286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.634254 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:58.713865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.713832 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-kube-api-access-8m9pn\" (UniqueName: \"kubernetes.io/projected/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-kube-api-access-8m9pn\") pod \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " Oct 30 01:21:58.714081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.713879 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-maas-api-tls\") pod \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " Oct 30 01:21:58.714081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.713973 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-openshift-service-ca\") pod \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " Oct 30 01:21:58.714081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.714041 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-metrics-certs\") pod \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\" (UID: \"aa6bfebb-1bf7-4279-82c8-bd98ec90d134\") " Oct 30 01:21:58.714423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.714382 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-openshift-service-ca" pod "aa6bfebb-1bf7-4279-82c8-bd98ec90d134" (UID: "aa6bfebb-1bf7-4279-82c8-bd98ec90d134"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:58.714731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.714665 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:58.716490 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.716462 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-maas-api-tls" pod "aa6bfebb-1bf7-4279-82c8-bd98ec90d134" (UID: "aa6bfebb-1bf7-4279-82c8-bd98ec90d134"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:58.716591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.716498 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-kube-api-access-8m9pn" pod "aa6bfebb-1bf7-4279-82c8-bd98ec90d134" (UID: "aa6bfebb-1bf7-4279-82c8-bd98ec90d134"). InnerVolumeSpecName "kube-api-access-8m9pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:58.716635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.716608 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-metrics-certs" pod "aa6bfebb-1bf7-4279-82c8-bd98ec90d134" (UID: "aa6bfebb-1bf7-4279-82c8-bd98ec90d134"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:58.756732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.753245 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:58.828344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.825437 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx"] Oct 30 01:21:58.828344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.825733 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" containerID="cri-o://9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda" gracePeriod=30 Oct 30 01:21:58.828344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.828259 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:58.828344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.828292 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9pn\" (UniqueName: \"kubernetes.io/projected/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-kube-api-access-8m9pn\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:58.828344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.828308 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6bfebb-1bf7-4279-82c8-bd98ec90d134-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:21:58.879497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:58.879349 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:59.048145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:59.046844 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:59.344861 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:59.344820 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:59.559384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:59.559354 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" Oct 30 01:21:59.559651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:59.559349 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j" event={"ID":"aa6bfebb-1bf7-4279-82c8-bd98ec90d134","Type":"ContainerDied","Data":"66473d560dd8ae9018d4f9cee19573d32e2decdf8e81ee27f1599e2466b58bda"} Oct 30 01:21:59.559651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:59.559491 4517 scope.go:122] "RemoveContainer" containerID="f6ad25c0c492fc00ac2c0e617b8464c0ac0f0007a98e1f64b43b850a2e13c24e" Oct 30 01:21:59.616672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:59.616527 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j"] Oct 30 01:21:59.621372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:21:59.621326 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-62f5d1-868f65f5cd-gw98j"] Oct 30 01:22:00.187592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.187531 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5"] Oct 30 01:22:00.188454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188415 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1885f34e-6420-4b73-aab4-18d173a0f95a" containerName="payload-processing" Oct 30 01:22:00.188454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188438 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1885f34e-6420-4b73-aab4-18d173a0f95a" containerName="payload-processing" Oct 30 01:22:00.188454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188458 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ba845c42-6120-4c91-b619-5d03dd41393e" containerName="payload-pre-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188478 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba845c42-6120-4c91-b619-5d03dd41393e" containerName="payload-pre-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188507 4517 cpu_manager.go:414] "containerMap: removing container" podUID="920d52a7-a474-44e2-b737-e6ff1412df15" containerName="revoke-keys" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188515 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920d52a7-a474-44e2-b737-e6ff1412df15" containerName="revoke-keys" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188540 4517 cpu_manager.go:414] "containerMap: removing container" podUID="25333518-39ac-4e6c-a406-80a09d216b14" containerName="payload-pre-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188547 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25333518-39ac-4e6c-a406-80a09d216b14" containerName="payload-pre-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188556 4517 cpu_manager.go:414] "containerMap: removing container" podUID="01bd6e21-5e90-4063-956d-09f0b7523bd2" containerName="payload-pre-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188563 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01bd6e21-5e90-4063-956d-09f0b7523bd2" containerName="payload-pre-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188580 4517 cpu_manager.go:414] "containerMap: removing container" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188587 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188601 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1bb00a4d-eb99-4bc2-a152-ec99247ac45c" containerName="payload-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188608 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb00a4d-eb99-4bc2-a152-ec99247ac45c" containerName="payload-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188624 4517 cpu_manager.go:414] "containerMap: removing container" podUID="c485ddbf-39d0-4c2f-b45c-4900a9775566" containerName="payload-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188630 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c485ddbf-39d0-4c2f-b45c-4900a9775566" containerName="payload-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188641 4517 cpu_manager.go:414] "containerMap: removing container" podUID="aa6bfebb-1bf7-4279-82c8-bd98ec90d134" containerName="maas-api" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188648 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6bfebb-1bf7-4279-82c8-bd98ec90d134" containerName="maas-api" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188826 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba845c42-6120-4c91-b619-5d03dd41393e" containerName="payload-pre-processing" Oct 30 01:22:00.188835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188844 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="920d52a7-a474-44e2-b737-e6ff1412df15" containerName="revoke-keys" Oct 30 01:22:00.189495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188860 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="25333518-39ac-4e6c-a406-80a09d216b14" containerName="payload-pre-processing" Oct 30 01:22:00.189495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188869 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="01bd6e21-5e90-4063-956d-09f0b7523bd2" containerName="payload-pre-processing" Oct 30 01:22:00.189495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188878 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="920fe4a0-48c7-4b43-a746-0543f8b3e9bf" containerName="istio-proxy" Oct 30 01:22:00.189495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188887 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb00a4d-eb99-4bc2-a152-ec99247ac45c" containerName="payload-processing" Oct 30 01:22:00.189495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188896 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="c485ddbf-39d0-4c2f-b45c-4900a9775566" containerName="payload-processing" Oct 30 01:22:00.189495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188904 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6bfebb-1bf7-4279-82c8-bd98ec90d134" containerName="maas-api" Oct 30 01:22:00.189495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.188914 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1885f34e-6420-4b73-aab4-18d173a0f95a" containerName="payload-processing" Oct 30 01:22:00.193960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.193915 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.197415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.197376 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5"] Oct 30 01:22:00.208264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.208231 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:22:00.218200 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.218168 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:22:00.239608 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.239571 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6bfebb-1bf7-4279-82c8-bd98ec90d134" path="/var/lib/kubelet/pods/aa6bfebb-1bf7-4279-82c8-bd98ec90d134/volumes" Oct 30 01:22:00.240272 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.240244 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b867f4-9150-4165-b3f1-6981161a32d6" path="/var/lib/kubelet/pods/e5b867f4-9150-4165-b3f1-6981161a32d6/volumes" Oct 30 01:22:00.262468 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.262417 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.262667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.262624 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwz2g\" (UniqueName: \"kubernetes.io/projected/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-kube-api-access-hwz2g\") pod \"maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.262749 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.262663 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.364756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.364663 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.364952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.364849 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwz2g\" (UniqueName: \"kubernetes.io/projected/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-kube-api-access-hwz2g\") pod \"maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.364952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.364880 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.365486 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.365465 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.367731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.367683 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.389350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.389310 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwz2g\" (UniqueName: \"kubernetes.io/projected/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-kube-api-access-hwz2g\") pod \"maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.516583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.516478 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:00.660622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.660579 4517 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:22:00.660826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.660654 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:22:00.762117 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:22:00.762076 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:22:00.834770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:00.834735 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5"] Oct 30 01:22:00.843287 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:22:00.843250 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1526ed0_b75c_44bb_b0cb_c8cdabe21b6b.slice/crio-8058103d1e8675d0cd442d6a94943cac9e91702cfd963acb27a196b8fd78d456 WatchSource:0}: Error finding container 8058103d1e8675d0cd442d6a94943cac9e91702cfd963acb27a196b8fd78d456: Status 404 returned error can't find the container with id 8058103d1e8675d0cd442d6a94943cac9e91702cfd963acb27a196b8fd78d456 Oct 30 01:22:01.263630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:01.263587 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:01.585359 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:01.585314 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" event={"ID":"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b","Type":"ContainerStarted","Data":"d1307271bdac0c67319bca3d09989f2e19529eea09eebceae977c8861a7811ae"} Oct 30 01:22:01.585613 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:01.585395 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" event={"ID":"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b","Type":"ContainerStarted","Data":"8058103d1e8675d0cd442d6a94943cac9e91702cfd963acb27a196b8fd78d456"} Oct 30 01:22:01.623728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:01.623654 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" podStartSLOduration=1.6236362610000001 podStartE2EDuration="1.623636261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:01.621483168 +0000 UTC m=+2859.533610307" watchObservedRunningTime="2026-10-30 01:22:01.623636261 +0000 UTC m=+2859.535763402" Oct 30 01:22:02.533255 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:02.533215 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:02.596125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:02.596084 4517 generic.go:356] "Generic (PLEG): container finished" podID="b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b" containerID="d1307271bdac0c67319bca3d09989f2e19529eea09eebceae977c8861a7811ae" exitCode=0 Oct 30 01:22:02.596340 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:02.596175 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" event={"ID":"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b","Type":"ContainerDied","Data":"d1307271bdac0c67319bca3d09989f2e19529eea09eebceae977c8861a7811ae"} Oct 30 01:22:02.699734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:02.691395 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:03.682139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:03.682099 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:03.838241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:03.838202 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:03.935018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:03.934965 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-kube-api-access-hwz2g\" (UniqueName: \"kubernetes.io/projected/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-kube-api-access-hwz2g\") pod \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " Oct 30 01:22:03.935210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:03.935132 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-openshift-ca\") pod \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " Oct 30 01:22:03.935210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:03.935199 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-xks-ca\") pod \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\" (UID: \"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b\") " Oct 30 01:22:03.935624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:03.935586 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-openshift-ca" pod "b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b" (UID: "b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:03.935941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:03.935916 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:03.937517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:03.937491 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-xks-ca" pod "b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b" (UID: "b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:03.937717 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:03.937682 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-kube-api-access-hwz2g" pod "b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b" (UID: "b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b"). InnerVolumeSpecName "kube-api-access-hwz2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:04.038807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.038695 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.039024 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.038834 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwz2g\" (UniqueName: \"kubernetes.io/projected/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b-kube-api-access-hwz2g\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.129401 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.129369 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:22:04.240171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.240123 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-certs\") pod \"870e23f2-fd01-4f9d-9c43-3b570733b19b\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " Oct 30 01:22:04.240497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.240464 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-credential-socket\") pod \"870e23f2-fd01-4f9d-9c43-3b570733b19b\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " Oct 30 01:22:04.241100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.240561 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870e23f2-fd01-4f9d-9c43-3b570733b19b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e23f2-fd01-4f9d-9c43-3b570733b19b-istiod-ca-cert\") pod \"870e23f2-fd01-4f9d-9c43-3b570733b19b\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " Oct 30 01:22:04.241100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.240609 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-podinfo\") pod \"870e23f2-fd01-4f9d-9c43-3b570733b19b\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " Oct 30 01:22:04.241100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.240828 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-data\") pod \"870e23f2-fd01-4f9d-9c43-3b570733b19b\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " Oct 30 01:22:04.241100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.240879 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-socket\") pod \"870e23f2-fd01-4f9d-9c43-3b570733b19b\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " Oct 30 01:22:04.241100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.240924 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-credential-socket" pod "870e23f2-fd01-4f9d-9c43-3b570733b19b" (UID: "870e23f2-fd01-4f9d-9c43-3b570733b19b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.241100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.240953 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-envoy\") pod \"870e23f2-fd01-4f9d-9c43-3b570733b19b\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " Oct 30 01:22:04.241100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.241001 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-token\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-token\") pod \"870e23f2-fd01-4f9d-9c43-3b570733b19b\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " Oct 30 01:22:04.241100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.241083 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-kube-api-access-9b6hv\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-kube-api-access-9b6hv\") pod \"870e23f2-fd01-4f9d-9c43-3b570733b19b\" (UID: \"870e23f2-fd01-4f9d-9c43-3b570733b19b\") " Oct 30 01:22:04.242649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.241255 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-certs" pod "870e23f2-fd01-4f9d-9c43-3b570733b19b" (UID: "870e23f2-fd01-4f9d-9c43-3b570733b19b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.242649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.241759 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.242649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.241781 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.242649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.242085 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870e23f2-fd01-4f9d-9c43-3b570733b19b-istiod-ca-cert" pod "870e23f2-fd01-4f9d-9c43-3b570733b19b" (UID: "870e23f2-fd01-4f9d-9c43-3b570733b19b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:04.242649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.242486 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-socket" pod "870e23f2-fd01-4f9d-9c43-3b570733b19b" (UID: "870e23f2-fd01-4f9d-9c43-3b570733b19b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.246460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.243800 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-data" pod "870e23f2-fd01-4f9d-9c43-3b570733b19b" (UID: "870e23f2-fd01-4f9d-9c43-3b570733b19b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.247399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.247363 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-kube-api-access-9b6hv" pod "870e23f2-fd01-4f9d-9c43-3b570733b19b" (UID: "870e23f2-fd01-4f9d-9c43-3b570733b19b"). InnerVolumeSpecName "kube-api-access-9b6hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:04.247644 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.247623 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-token" pod "870e23f2-fd01-4f9d-9c43-3b570733b19b" (UID: "870e23f2-fd01-4f9d-9c43-3b570733b19b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:04.248525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.248502 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-envoy" pod "870e23f2-fd01-4f9d-9c43-3b570733b19b" (UID: "870e23f2-fd01-4f9d-9c43-3b570733b19b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.249329 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.249300 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-podinfo" pod "870e23f2-fd01-4f9d-9c43-3b570733b19b" (UID: "870e23f2-fd01-4f9d-9c43-3b570733b19b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:04.344738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.328502 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt"] Oct 30 01:22:04.344738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.331902 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb"] Oct 30 01:22:04.344738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.343148 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.344738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.343187 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.344738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.343201 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.344738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.343214 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.344738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.343229 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b6hv\" (UniqueName: \"kubernetes.io/projected/870e23f2-fd01-4f9d-9c43-3b570733b19b-kube-api-access-9b6hv\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.344738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.343248 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e23f2-fd01-4f9d-9c43-3b570733b19b-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.344738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.343262 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e23f2-fd01-4f9d-9c43-3b570733b19b-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.414123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.414031 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:04.613449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.613414 4517 generic.go:356] "Generic (PLEG): container finished" podID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerID="9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda" exitCode=0 Oct 30 01:22:04.613624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.613487 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" Oct 30 01:22:04.613624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.613510 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" event={"ID":"870e23f2-fd01-4f9d-9c43-3b570733b19b","Type":"ContainerDied","Data":"9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda"} Oct 30 01:22:04.613624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.613579 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx" event={"ID":"870e23f2-fd01-4f9d-9c43-3b570733b19b","Type":"ContainerDied","Data":"9fc7b2e25ed43b71477ecf00d81c38a701c0cdd4f101b5e52fced23bedf770f4"} Oct 30 01:22:04.613624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.613613 4517 scope.go:122] "RemoveContainer" containerID="9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda" Oct 30 01:22:04.615353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.615325 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" Oct 30 01:22:04.615516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.615326 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5" event={"ID":"b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b","Type":"ContainerDied","Data":"8058103d1e8675d0cd442d6a94943cac9e91702cfd963acb27a196b8fd78d456"} Oct 30 01:22:04.615516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.615472 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8058103d1e8675d0cd442d6a94943cac9e91702cfd963acb27a196b8fd78d456" Oct 30 01:22:04.626752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.626686 4517 scope.go:122] "RemoveContainer" containerID="9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda" Oct 30 01:22:04.627179 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:22:04.627153 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda\": container with ID starting with 9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda not found: ID does not exist" containerID="9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda" Oct 30 01:22:04.627254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.627193 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda"} err="failed to get container status \"9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda\": rpc error: code = NotFound desc = could not find container \"9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda\": container with ID starting with 9b98b59763c2578a57a18a73d3b326bada6ea3bb1c5c3804fe42dffd7dcdcfda not found: ID does not exist" Oct 30 01:22:04.674866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.674772 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx"] Oct 30 01:22:04.684000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:04.683960 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2thxx"] Oct 30 01:22:05.090867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:05.090833 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:05.336118 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:05.325396 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:06.195543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:06.195509 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:06.259138 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:06.259084 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" path="/var/lib/kubelet/pods/870e23f2-fd01-4f9d-9c43-3b570733b19b/volumes" Oct 30 01:22:06.325664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:06.325629 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:06.449307 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:06.449174 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:06.559415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:06.559376 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:06.907155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:06.907117 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:07.040684 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:07.040650 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:08.476819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:08.476786 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:10.520101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:10.520064 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:10.644652 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:10.644600 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:10.764165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:10.764125 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:11.902557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:11.902514 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx"] Oct 30 01:22:12.010067 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:12.010026 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:12.350836 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:12.350802 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:12.467923 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:12.467883 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:12.866597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:12.866495 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:14.655932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:14.655888 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:16.586006 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:16.585940 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:17.659395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:17.659362 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:17.781933 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:17.781879 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:18.231431 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:18.231367 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:19.023108 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.023074 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:19.371270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.371215 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" containerID="cri-o://d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef" gracePeriod=45 Oct 30 01:22:19.384197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.384161 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" containerID="cri-o://5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0" gracePeriod=45 Oct 30 01:22:19.730235 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.730207 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 01:22:19.734288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.734259 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 01:22:19.887137 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887098 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tmp-dir\") pod \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " Oct 30 01:22:19.887399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887141 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tmp-dir\") pod \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " Oct 30 01:22:19.887399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887232 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-home\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-home\") pod \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " Oct 30 01:22:19.887399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887262 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tls-certs\") pod \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " Oct 30 01:22:19.887399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887298 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-kube-api-access-xwvmp\" (UniqueName: \"kubernetes.io/projected/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-kube-api-access-xwvmp\") pod \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " Oct 30 01:22:19.887399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887331 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-dshm\") pod \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " Oct 30 01:22:19.887399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887379 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-home\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-home\") pod \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " Oct 30 01:22:19.887696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887441 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-model-cache\") pod \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " Oct 30 01:22:19.887696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887472 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tls-certs\") pod \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " Oct 30 01:22:19.887696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887597 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-model-cache\") pod \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\" (UID: \"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9\") " Oct 30 01:22:19.887696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887660 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-kube-api-access-2r4dx\" (UniqueName: \"kubernetes.io/projected/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-kube-api-access-2r4dx\") pod \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " Oct 30 01:22:19.887961 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887696 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-dshm\") pod \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\" (UID: \"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e\") " Oct 30 01:22:19.888191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.887969 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tmp-dir" pod "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" (UID: "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.888403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.888379 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tmp-dir" pod "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" (UID: "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.888511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.888455 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.888511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.888464 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-home" pod "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" (UID: "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.888511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.888481 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-model-cache" pod "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" (UID: "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.888746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.888693 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-model-cache" pod "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" (UID: "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.888832 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.888760 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-home" pod "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" (UID: "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.891180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.891148 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-dshm" pod "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" (UID: "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.892931 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.892904 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-kube-api-access-2r4dx" pod "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" (UID: "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e"). InnerVolumeSpecName "kube-api-access-2r4dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:19.893351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.893328 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tls-certs" pod "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" (UID: "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:19.896065 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.896032 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tls-certs" pod "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" (UID: "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:19.896298 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.896272 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-kube-api-access-xwvmp" pod "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" (UID: "f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9"). InnerVolumeSpecName "kube-api-access-xwvmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:19.897607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.897577 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-dshm" pod "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" (UID: "aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.940002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.939909 4517 generic.go:356] "Generic (PLEG): container finished" podID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerID="d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef" exitCode=0 Oct 30 01:22:19.940002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.939954 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" Oct 30 01:22:19.940002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.939989 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" event={"ID":"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9","Type":"ContainerDied","Data":"d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef"} Oct 30 01:22:19.940311 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.940030 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" event={"ID":"f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9","Type":"ContainerDied","Data":"0d3b0a5b9b271f5a75d2fad1b1f6571a4b068d3908d9baece833b9510dff5950"} Oct 30 01:22:19.940311 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.940050 4517 scope.go:122] "RemoveContainer" containerID="d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef" Oct 30 01:22:19.941776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.941744 4517 generic.go:356] "Generic (PLEG): container finished" podID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerID="5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0" exitCode=0 Oct 30 01:22:19.941887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.941803 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" Oct 30 01:22:19.941936 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.941798 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" event={"ID":"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e","Type":"ContainerDied","Data":"5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0"} Oct 30 01:22:19.942070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.941925 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb" event={"ID":"aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e","Type":"ContainerDied","Data":"79aad0b655fa43822ac7c05622b93d347e13628524535281b729e2da931f9d28"} Oct 30 01:22:19.952831 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.952813 4517 scope.go:122] "RemoveContainer" containerID="d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef" Oct 30 01:22:19.953152 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:22:19.953130 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef\": container with ID starting with d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef not found: ID does not exist" containerID="d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef" Oct 30 01:22:19.953268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.953159 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef"} err="failed to get container status \"d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef\": rpc error: code = NotFound desc = could not find container \"d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef\": container with ID starting with d6e096a493d7387ba3c2a983cbb687091ce3c285420092dfb60b8ebd35b599ef not found: ID does not exist" Oct 30 01:22:19.953268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.953177 4517 scope.go:122] "RemoveContainer" containerID="5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0" Oct 30 01:22:19.965194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.965171 4517 scope.go:122] "RemoveContainer" containerID="5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0" Oct 30 01:22:19.965600 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:22:19.965557 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0\": container with ID starting with 5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0 not found: ID does not exist" containerID="5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0" Oct 30 01:22:19.965746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.965604 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0"} err="failed to get container status \"5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0\": rpc error: code = NotFound desc = could not find container \"5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0\": container with ID starting with 5ed8a6f9a9d298f102dcd974ec761ccbbd14f483cd524809f660c723d1b08cb0 not found: ID does not exist" Oct 30 01:22:19.987242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.987186 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb"] Oct 30 01:22:19.990192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990154 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990192 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990192 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990210 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwvmp\" (UniqueName: \"kubernetes.io/projected/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-kube-api-access-xwvmp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990222 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990233 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990246 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990256 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990267 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990278 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r4dx\" (UniqueName: \"kubernetes.io/projected/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-kube-api-access-2r4dx\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990289 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.990436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.990303 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.994569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:19.994529 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/simulator-w4-6add35-kserve-b78c678d-7bffb"] Oct 30 01:22:20.025753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:20.025670 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt"] Oct 30 01:22:20.031275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:20.031238 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt"] Oct 30 01:22:20.067111 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:20.067082 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:20.235237 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:20.235156 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" path="/var/lib/kubelet/pods/aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e/volumes" Oct 30 01:22:20.235666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:20.235652 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" path="/var/lib/kubelet/pods/f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9/volumes" Oct 30 01:22:20.562697 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:20.562647 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/premium-w4-6add35-kserve-d54c49f65-vhnwt" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:22:22.356315 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:22.356280 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:23.928071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:23.928020 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:25.735423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:25.735376 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:26.603346 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:26.603297 4517 scope.go:122] "RemoveContainer" containerID="3a4f6c81a36a27695bf90cc2101a037c0109bba926a1bdddea78d49ccb98a143" Oct 30 01:22:26.615245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:26.615220 4517 scope.go:122] "RemoveContainer" containerID="10b9f41b7dac3742a1a08b810d2d5ca341742aeda01f2c96f20c14a32332728b" Oct 30 01:22:26.930616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:26.930519 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" containerID="cri-o://c6e72baa4e914436896014d83e9eb04ccdd0d5dd0cd7a6560b33609dc6c67a34" gracePeriod=45 Oct 30 01:22:27.007233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.007165 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": dial tcp 10.133.0.39:8000: connect: connection refused" Oct 30 01:22:27.057020 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.056984 4517 generic.go:356] "Generic (PLEG): container finished" podID="731830bd-e885-4e72-a650-6682c5aa029b" containerID="c6e72baa4e914436896014d83e9eb04ccdd0d5dd0cd7a6560b33609dc6c67a34" exitCode=0 Oct 30 01:22:27.057203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.057054 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" event={"ID":"731830bd-e885-4e72-a650-6682c5aa029b","Type":"ContainerDied","Data":"c6e72baa4e914436896014d83e9eb04ccdd0d5dd0cd7a6560b33609dc6c67a34"} Oct 30 01:22:27.234405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.234375 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 01:22:27.400269 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.400226 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-dshm\") pod \"731830bd-e885-4e72-a650-6682c5aa029b\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " Oct 30 01:22:27.400450 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.400355 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-home\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-home\") pod \"731830bd-e885-4e72-a650-6682c5aa029b\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " Oct 30 01:22:27.400533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.400478 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-tmp-dir\") pod \"731830bd-e885-4e72-a650-6682c5aa029b\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " Oct 30 01:22:27.400533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.400517 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731830bd-e885-4e72-a650-6682c5aa029b-kube-api-access-8dpx9\" (UniqueName: \"kubernetes.io/projected/731830bd-e885-4e72-a650-6682c5aa029b-kube-api-access-8dpx9\") pod \"731830bd-e885-4e72-a650-6682c5aa029b\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " Oct 30 01:22:27.400673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.400564 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731830bd-e885-4e72-a650-6682c5aa029b-tls-certs\" (UniqueName: \"kubernetes.io/secret/731830bd-e885-4e72-a650-6682c5aa029b-tls-certs\") pod \"731830bd-e885-4e72-a650-6682c5aa029b\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " Oct 30 01:22:27.400861 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.400829 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-tmp-dir" pod "731830bd-e885-4e72-a650-6682c5aa029b" (UID: "731830bd-e885-4e72-a650-6682c5aa029b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.400954 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.400920 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-home" pod "731830bd-e885-4e72-a650-6682c5aa029b" (UID: "731830bd-e885-4e72-a650-6682c5aa029b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.401145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.401120 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-model-cache\") pod \"731830bd-e885-4e72-a650-6682c5aa029b\" (UID: \"731830bd-e885-4e72-a650-6682c5aa029b\") " Oct 30 01:22:27.401618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.401581 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-model-cache" pod "731830bd-e885-4e72-a650-6682c5aa029b" (UID: "731830bd-e885-4e72-a650-6682c5aa029b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.402803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.402779 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.402999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.402980 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.403067 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.403005 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.403774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.403657 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-dshm" pod "731830bd-e885-4e72-a650-6682c5aa029b" (UID: "731830bd-e885-4e72-a650-6682c5aa029b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.404935 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.404906 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731830bd-e885-4e72-a650-6682c5aa029b-kube-api-access-8dpx9" pod "731830bd-e885-4e72-a650-6682c5aa029b" (UID: "731830bd-e885-4e72-a650-6682c5aa029b"). InnerVolumeSpecName "kube-api-access-8dpx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:27.405161 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.405132 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731830bd-e885-4e72-a650-6682c5aa029b-tls-certs" pod "731830bd-e885-4e72-a650-6682c5aa029b" (UID: "731830bd-e885-4e72-a650-6682c5aa029b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:27.452613 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.452522 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:27.527622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.527536 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731830bd-e885-4e72-a650-6682c5aa029b-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.527622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.527563 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpx9\" (UniqueName: \"kubernetes.io/projected/731830bd-e885-4e72-a650-6682c5aa029b-kube-api-access-8dpx9\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.527622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:27.527573 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731830bd-e885-4e72-a650-6682c5aa029b-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:28.067841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:28.067809 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" Oct 30 01:22:28.068369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:28.067808 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx" event={"ID":"731830bd-e885-4e72-a650-6682c5aa029b","Type":"ContainerDied","Data":"5276f9ee636ec619da1a37f208b6e9f9cdd0c28bdecef29b4121c89d0be66953"} Oct 30 01:22:28.068369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:28.067951 4517 scope.go:122] "RemoveContainer" containerID="c6e72baa4e914436896014d83e9eb04ccdd0d5dd0cd7a6560b33609dc6c67a34" Oct 30 01:22:28.115516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:28.115476 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx"] Oct 30 01:22:28.125592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:28.125564 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/simulator-w0-2b4105-kserve-54755c796c-qwwvx"] Oct 30 01:22:28.235645 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:28.235607 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731830bd-e885-4e72-a650-6682c5aa029b" path="/var/lib/kubelet/pods/731830bd-e885-4e72-a650-6682c5aa029b/volumes" Oct 30 01:22:29.177564 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:29.177521 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:30.761856 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:22:30.761804 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:22:31.134590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.134551 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l"] Oct 30 01:22:31.135390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135365 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" Oct 30 01:22:31.135390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135391 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" Oct 30 01:22:31.135648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135417 4517 cpu_manager.go:414] "containerMap: removing container" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" Oct 30 01:22:31.135648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135424 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" Oct 30 01:22:31.135648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135439 4517 cpu_manager.go:414] "containerMap: removing container" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" Oct 30 01:22:31.135648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135446 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" Oct 30 01:22:31.135648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135475 4517 cpu_manager.go:414] "containerMap: removing container" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" Oct 30 01:22:31.135648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135483 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" Oct 30 01:22:31.135648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135504 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b" containerName="revoke-keys" Oct 30 01:22:31.135648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135513 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b" containerName="revoke-keys" Oct 30 01:22:31.136032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135656 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="870e23f2-fd01-4f9d-9c43-3b570733b19b" containerName="istio-proxy" Oct 30 01:22:31.136032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135671 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b" containerName="revoke-keys" Oct 30 01:22:31.136032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135682 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ed8efc-1cab-412b-a7d7-9ba1899d6ce9" containerName="main" Oct 30 01:22:31.136032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135693 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2e4e6c-f6d1-4d9e-866b-7504d3a17f5e" containerName="main" Oct 30 01:22:31.136032 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.135701 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="731830bd-e885-4e72-a650-6682c5aa029b" containerName="main" Oct 30 01:22:31.141407 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.141373 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.158790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.158753 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l"] Oct 30 01:22:31.176997 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.176955 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1927f3f-lwt6l\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.177222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.177145 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1927f3f-lwt6l\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.177222 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.177205 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pww5w\" (UniqueName: \"kubernetes.io/projected/af096e76-cc77-421f-a309-e238ccc03f62-kube-api-access-pww5w\") pod \"maas-api-revoke-sub-d1927f3f-lwt6l\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.234459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.234408 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:22:31.234664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.234486 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:22:31.279122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.279081 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1927f3f-lwt6l\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.279345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.279199 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1927f3f-lwt6l\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.279345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.279234 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pww5w\" (UniqueName: \"kubernetes.io/projected/af096e76-cc77-421f-a309-e238ccc03f62-kube-api-access-pww5w\") pod \"maas-api-revoke-sub-d1927f3f-lwt6l\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.279989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.279958 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1927f3f-lwt6l\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.282074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.282046 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1927f3f-lwt6l\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.303507 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.303468 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pww5w\" (UniqueName: \"kubernetes.io/projected/af096e76-cc77-421f-a309-e238ccc03f62-kube-api-access-pww5w\") pod \"maas-api-revoke-sub-d1927f3f-lwt6l\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.463387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.463291 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:31.622880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.622841 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:31.787166 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:31.787128 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l"] Oct 30 01:22:31.790665 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:22:31.790625 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf096e76_cc77_421f_a309_e238ccc03f62.slice/crio-3c1cabf17382579ef1e006035b8c0834f314ba0678867cd8408ec216e094d7b4 WatchSource:0}: Error finding container 3c1cabf17382579ef1e006035b8c0834f314ba0678867cd8408ec216e094d7b4: Status 404 returned error can't find the container with id 3c1cabf17382579ef1e006035b8c0834f314ba0678867cd8408ec216e094d7b4 Oct 30 01:22:32.104497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:32.104448 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" event={"ID":"af096e76-cc77-421f-a309-e238ccc03f62","Type":"ContainerStarted","Data":"1e3e77201d00fd24c281d5fe0eae16e1df09e011716a4f8aeff7b712ce569178"} Oct 30 01:22:32.104679 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:32.104528 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" event={"ID":"af096e76-cc77-421f-a309-e238ccc03f62","Type":"ContainerStarted","Data":"3c1cabf17382579ef1e006035b8c0834f314ba0678867cd8408ec216e094d7b4"} Oct 30 01:22:32.145194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:32.145134 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" podStartSLOduration=1.145118357 podStartE2EDuration="1.145118357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:32.142059668 +0000 UTC m=+2890.054186809" watchObservedRunningTime="2026-10-30 01:22:32.145118357 +0000 UTC m=+2890.057245498" Oct 30 01:22:33.115922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:33.115875 4517 generic.go:356] "Generic (PLEG): container finished" podID="af096e76-cc77-421f-a309-e238ccc03f62" containerID="1e3e77201d00fd24c281d5fe0eae16e1df09e011716a4f8aeff7b712ce569178" exitCode=0 Oct 30 01:22:33.116423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:33.115970 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" event={"ID":"af096e76-cc77-421f-a309-e238ccc03f62","Type":"ContainerDied","Data":"1e3e77201d00fd24c281d5fe0eae16e1df09e011716a4f8aeff7b712ce569178"} Oct 30 01:22:33.183791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:33.183748 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:34.305667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.305641 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:34.443559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.443467 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-xks-ca\") pod \"af096e76-cc77-421f-a309-e238ccc03f62\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " Oct 30 01:22:34.443559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.443545 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-openshift-ca\") pod \"af096e76-cc77-421f-a309-e238ccc03f62\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " Oct 30 01:22:34.443856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.443624 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af096e76-cc77-421f-a309-e238ccc03f62-kube-api-access-pww5w\" (UniqueName: \"kubernetes.io/projected/af096e76-cc77-421f-a309-e238ccc03f62-kube-api-access-pww5w\") pod \"af096e76-cc77-421f-a309-e238ccc03f62\" (UID: \"af096e76-cc77-421f-a309-e238ccc03f62\") " Oct 30 01:22:34.443978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.443952 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-openshift-ca" pod "af096e76-cc77-421f-a309-e238ccc03f62" (UID: "af096e76-cc77-421f-a309-e238ccc03f62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:34.444279 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.444262 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:34.445904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.445864 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-xks-ca" pod "af096e76-cc77-421f-a309-e238ccc03f62" (UID: "af096e76-cc77-421f-a309-e238ccc03f62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:34.446043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.445996 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af096e76-cc77-421f-a309-e238ccc03f62-kube-api-access-pww5w" pod "af096e76-cc77-421f-a309-e238ccc03f62" (UID: "af096e76-cc77-421f-a309-e238ccc03f62"). InnerVolumeSpecName "kube-api-access-pww5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:34.546077 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.546022 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pww5w\" (UniqueName: \"kubernetes.io/projected/af096e76-cc77-421f-a309-e238ccc03f62-kube-api-access-pww5w\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:34.546077 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.546066 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af096e76-cc77-421f-a309-e238ccc03f62-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:34.746992 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:34.746885 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:35.135598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:35.135562 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" Oct 30 01:22:35.135851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:35.135596 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l" event={"ID":"af096e76-cc77-421f-a309-e238ccc03f62","Type":"ContainerDied","Data":"3c1cabf17382579ef1e006035b8c0834f314ba0678867cd8408ec216e094d7b4"} Oct 30 01:22:35.135851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:35.135652 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c1cabf17382579ef1e006035b8c0834f314ba0678867cd8408ec216e094d7b4" Oct 30 01:22:36.235421 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.235388 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:36.313509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.313468 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5"] Oct 30 01:22:36.313745 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.313697 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" podUID="2183c7f2-b9bf-4caa-9761-249a6736dc59" containerName="payload-processing" containerID="cri-o://7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b" gracePeriod=30 Oct 30 01:22:36.354986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.354929 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg"] Oct 30 01:22:36.355462 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.355423 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" podUID="73bd7b76-ad2a-47a6-8618-6d70ffd42ca1" containerName="payload-pre-processing" containerID="cri-o://58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560" gracePeriod=30 Oct 30 01:22:36.413577 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.413533 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:36.565999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.565960 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:36.753816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.753744 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:22:36.762840 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.762812 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:22:36.768424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.768396 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:36.890205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.889974 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2183c7f2-b9bf-4caa-9761-249a6736dc59-kube-api-access-l8dbt\" (UniqueName: \"kubernetes.io/projected/2183c7f2-b9bf-4caa-9761-249a6736dc59-kube-api-access-l8dbt\") pod \"2183c7f2-b9bf-4caa-9761-249a6736dc59\" (UID: \"2183c7f2-b9bf-4caa-9761-249a6736dc59\") " Oct 30 01:22:36.890205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.890069 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-kube-api-access-jws4v\" (UniqueName: \"kubernetes.io/projected/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-kube-api-access-jws4v\") pod \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\" (UID: \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\") " Oct 30 01:22:36.890205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.890131 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2183c7f2-b9bf-4caa-9761-249a6736dc59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183c7f2-b9bf-4caa-9761-249a6736dc59-plugins-config-volume\") pod \"2183c7f2-b9bf-4caa-9761-249a6736dc59\" (UID: \"2183c7f2-b9bf-4caa-9761-249a6736dc59\") " Oct 30 01:22:36.890582 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.890225 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-plugins-config-volume\") pod \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\" (UID: \"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1\") " Oct 30 01:22:36.891515 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.891481 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-plugins-config-volume" pod "73bd7b76-ad2a-47a6-8618-6d70ffd42ca1" (UID: "73bd7b76-ad2a-47a6-8618-6d70ffd42ca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:36.894675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.894640 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:36.895485 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.895454 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2183c7f2-b9bf-4caa-9761-249a6736dc59-plugins-config-volume" pod "2183c7f2-b9bf-4caa-9761-249a6736dc59" (UID: "2183c7f2-b9bf-4caa-9761-249a6736dc59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:36.898096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.898059 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-kube-api-access-jws4v" pod "73bd7b76-ad2a-47a6-8618-6d70ffd42ca1" (UID: "73bd7b76-ad2a-47a6-8618-6d70ffd42ca1"). InnerVolumeSpecName "kube-api-access-jws4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:36.899263 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.899219 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2183c7f2-b9bf-4caa-9761-249a6736dc59-kube-api-access-l8dbt" pod "2183c7f2-b9bf-4caa-9761-249a6736dc59" (UID: "2183c7f2-b9bf-4caa-9761-249a6736dc59"). InnerVolumeSpecName "kube-api-access-l8dbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:36.992904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.992867 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8dbt\" (UniqueName: \"kubernetes.io/projected/2183c7f2-b9bf-4caa-9761-249a6736dc59-kube-api-access-l8dbt\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:36.992904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.992898 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jws4v\" (UniqueName: \"kubernetes.io/projected/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-kube-api-access-jws4v\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:36.992904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.992908 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2183c7f2-b9bf-4caa-9761-249a6736dc59-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:36.993254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:36.992917 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:37.010044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.009938 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:37.155938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.155900 4517 generic.go:356] "Generic (PLEG): container finished" podID="2183c7f2-b9bf-4caa-9761-249a6736dc59" containerID="7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b" exitCode=0 Oct 30 01:22:37.156167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.155936 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" event={"ID":"2183c7f2-b9bf-4caa-9761-249a6736dc59","Type":"ContainerDied","Data":"7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b"} Oct 30 01:22:37.156167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.155958 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" Oct 30 01:22:37.156167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.155994 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5" event={"ID":"2183c7f2-b9bf-4caa-9761-249a6736dc59","Type":"ContainerDied","Data":"ffd7d1d56ef7d003f9f2efa544e9a730e41e65067f114e697be8bd8d9b2672f0"} Oct 30 01:22:37.156167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.156021 4517 scope.go:122] "RemoveContainer" containerID="7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b" Oct 30 01:22:37.157762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.157734 4517 generic.go:356] "Generic (PLEG): container finished" podID="73bd7b76-ad2a-47a6-8618-6d70ffd42ca1" containerID="58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560" exitCode=0 Oct 30 01:22:37.157945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.157781 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" event={"ID":"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1","Type":"ContainerDied","Data":"58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560"} Oct 30 01:22:37.158074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.157975 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" event={"ID":"73bd7b76-ad2a-47a6-8618-6d70ffd42ca1","Type":"ContainerDied","Data":"dc288a442a45857070e234ca0ae3aabe02cd55f73b1493f204b292d7b9fc8dda"} Oct 30 01:22:37.158074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.157811 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg" Oct 30 01:22:37.174294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.174260 4517 scope.go:122] "RemoveContainer" containerID="7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b" Oct 30 01:22:37.175081 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:22:37.174778 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b\": container with ID starting with 7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b not found: ID does not exist" containerID="7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b" Oct 30 01:22:37.175081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.174830 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b"} err="failed to get container status \"7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b\": rpc error: code = NotFound desc = could not find container \"7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b\": container with ID starting with 7561856c04cb51968de55be78fba19fd8cd4745394b22c4beb918f0d785f9e9b not found: ID does not exist" Oct 30 01:22:37.175081 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.174853 4517 scope.go:122] "RemoveContainer" containerID="58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560" Oct 30 01:22:37.186097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.186071 4517 scope.go:122] "RemoveContainer" containerID="58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560" Oct 30 01:22:37.186477 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:22:37.186453 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560\": container with ID starting with 58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560 not found: ID does not exist" containerID="58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560" Oct 30 01:22:37.186518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.186490 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560"} err="failed to get container status \"58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560\": rpc error: code = NotFound desc = could not find container \"58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560\": container with ID starting with 58ac2de2389bd1589b309c6bb82c17597da569a6c4504417d8281a43a464d560 not found: ID does not exist" Oct 30 01:22:37.305134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.305062 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg"] Oct 30 01:22:37.350373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.350210 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:37.388736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.361273 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-db764875-k8wxg"] Oct 30 01:22:37.411620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.403642 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5"] Oct 30 01:22:37.445277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.445052 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38b47d-65c9d667c6-snwt5"] Oct 30 01:22:37.473194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.473146 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:37.565631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.565545 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5"] Oct 30 01:22:37.568666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.568642 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6cba49-6tcf5"] Oct 30 01:22:37.592427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:37.592391 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:38.059830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:38.059796 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:38.235966 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:38.235930 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2183c7f2-b9bf-4caa-9761-249a6736dc59" path="/var/lib/kubelet/pods/2183c7f2-b9bf-4caa-9761-249a6736dc59/volumes" Oct 30 01:22:38.236388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:38.236374 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bd7b76-ad2a-47a6-8618-6d70ffd42ca1" path="/var/lib/kubelet/pods/73bd7b76-ad2a-47a6-8618-6d70ffd42ca1/volumes" Oct 30 01:22:38.236807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:38.236793 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b" path="/var/lib/kubelet/pods/b1526ed0-b75c-44bb-b0cb-c8cdabe21b6b/volumes" Oct 30 01:22:40.130181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:40.130146 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.754110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:41.754076 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.867422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:41.867379 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.015966 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:42.015875 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.354730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:42.354680 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.693659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:42.693558 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:43.115058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:43.115019 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:43.743069 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:43.743037 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:44.391186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:44.391137 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:44.822459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:44.822388 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:44.955232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:44.955194 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:45.130430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:45.130330 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:45.139628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:45.139585 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz"] Oct 30 01:22:45.139941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:45.139909 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" containerID="cri-o://8b9c7140e4b3374c50c4a494c4266a6798859968c3374461d171b3d820183eec" gracePeriod=30 Oct 30 01:22:45.298736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:45.259378 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:45.472371 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:45.472238 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:45.608733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:45.608564 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.427920 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:47.427883 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.036751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:48.036687 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc"] Oct 30 01:22:48.046947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:48.046916 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41db4590-2pcrc"] Oct 30 01:22:48.248618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:48.248567 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947aa77f-e60c-40c9-b801-89155355e852" path="/var/lib/kubelet/pods/947aa77f-e60c-40c9-b801-89155355e852/volumes" Oct 30 01:22:48.810070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:48.810017 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.942666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:48.942562 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:49.170050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:49.169955 4517 patch_prober.go:29] interesting pod/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:22:49.170360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:49.170317 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:22:49.212675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:49.212634 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.061987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.061940 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.396276 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.396230 4517 generic.go:356] "Generic (PLEG): container finished" podID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerID="8b9c7140e4b3374c50c4a494c4266a6798859968c3374461d171b3d820183eec" exitCode=0 Oct 30 01:22:50.396567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.396485 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" event={"ID":"66d5f4d6-77bb-4000-b374-57eac9dced96","Type":"ContainerDied","Data":"8b9c7140e4b3374c50c4a494c4266a6798859968c3374461d171b3d820183eec"} Oct 30 01:22:50.396567 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.396551 4517 scope.go:122] "RemoveContainer" containerID="35b4b4c03a4cfe8e9e040be1a2a9f9e549d60db0e35d2c719de82373b5e6d1b9" Oct 30 01:22:50.453162 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.453125 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:22:50.528223 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.528182 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-certs\") pod \"66d5f4d6-77bb-4000-b374-57eac9dced96\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " Oct 30 01:22:50.528449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.528260 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-kube-api-access-r9gh8\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-kube-api-access-r9gh8\") pod \"66d5f4d6-77bb-4000-b374-57eac9dced96\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " Oct 30 01:22:50.528449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.528437 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-envoy\") pod \"66d5f4d6-77bb-4000-b374-57eac9dced96\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " Oct 30 01:22:50.528626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.528474 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-credential-socket\") pod \"66d5f4d6-77bb-4000-b374-57eac9dced96\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " Oct 30 01:22:50.528626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.528509 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-socket\") pod \"66d5f4d6-77bb-4000-b374-57eac9dced96\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " Oct 30 01:22:50.528626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.528598 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-data\") pod \"66d5f4d6-77bb-4000-b374-57eac9dced96\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " Oct 30 01:22:50.528868 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.528685 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-token\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-token\") pod \"66d5f4d6-77bb-4000-b374-57eac9dced96\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " Oct 30 01:22:50.528938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.528870 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-podinfo\") pod \"66d5f4d6-77bb-4000-b374-57eac9dced96\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " Oct 30 01:22:50.528938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.528923 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d5f4d6-77bb-4000-b374-57eac9dced96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d5f4d6-77bb-4000-b374-57eac9dced96-istiod-ca-cert\") pod \"66d5f4d6-77bb-4000-b374-57eac9dced96\" (UID: \"66d5f4d6-77bb-4000-b374-57eac9dced96\") " Oct 30 01:22:50.529376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.529348 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-data" pod "66d5f4d6-77bb-4000-b374-57eac9dced96" (UID: "66d5f4d6-77bb-4000-b374-57eac9dced96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:50.529779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.529755 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-credential-socket" pod "66d5f4d6-77bb-4000-b374-57eac9dced96" (UID: "66d5f4d6-77bb-4000-b374-57eac9dced96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:50.530197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.530174 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-socket" pod "66d5f4d6-77bb-4000-b374-57eac9dced96" (UID: "66d5f4d6-77bb-4000-b374-57eac9dced96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:50.530758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.530727 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d5f4d6-77bb-4000-b374-57eac9dced96-istiod-ca-cert" pod "66d5f4d6-77bb-4000-b374-57eac9dced96" (UID: "66d5f4d6-77bb-4000-b374-57eac9dced96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:50.530851 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.530810 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-certs" pod "66d5f4d6-77bb-4000-b374-57eac9dced96" (UID: "66d5f4d6-77bb-4000-b374-57eac9dced96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:50.531485 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.531461 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.531627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.531610 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.531782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.531763 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.531898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.531885 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d5f4d6-77bb-4000-b374-57eac9dced96-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.532002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.531989 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.532901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.532866 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-envoy" pod "66d5f4d6-77bb-4000-b374-57eac9dced96" (UID: "66d5f4d6-77bb-4000-b374-57eac9dced96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:50.533244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.533213 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-kube-api-access-r9gh8" pod "66d5f4d6-77bb-4000-b374-57eac9dced96" (UID: "66d5f4d6-77bb-4000-b374-57eac9dced96"). InnerVolumeSpecName "kube-api-access-r9gh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:50.533880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.533850 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-token" pod "66d5f4d6-77bb-4000-b374-57eac9dced96" (UID: "66d5f4d6-77bb-4000-b374-57eac9dced96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:50.536303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.536267 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-podinfo" pod "66d5f4d6-77bb-4000-b374-57eac9dced96" (UID: "66d5f4d6-77bb-4000-b374-57eac9dced96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:50.633555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.633463 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.633555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.633504 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.633555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.633514 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9gh8\" (UniqueName: \"kubernetes.io/projected/66d5f4d6-77bb-4000-b374-57eac9dced96-kube-api-access-r9gh8\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.633555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.633524 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d5f4d6-77bb-4000-b374-57eac9dced96-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.817555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:50.817470 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:51.034673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:51.034643 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:51.406977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:51.406884 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" Oct 30 01:22:51.406977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:51.406911 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz" event={"ID":"66d5f4d6-77bb-4000-b374-57eac9dced96","Type":"ContainerDied","Data":"cbc7275313fe1c2b37f4207ab9728e7cf7f016465a27918e00185122dc7c9e23"} Oct 30 01:22:51.407518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:51.406978 4517 scope.go:122] "RemoveContainer" containerID="8b9c7140e4b3374c50c4a494c4266a6798859968c3374461d171b3d820183eec" Oct 30 01:22:51.416244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:51.415916 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:51.495821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:51.495781 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz"] Oct 30 01:22:51.502201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:51.502152 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-38b47d-openshift-default-649dd76fb5-8m5tz"] Oct 30 01:22:51.527271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:51.527235 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:52.236939 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:52.236906 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" path="/var/lib/kubelet/pods/66d5f4d6-77bb-4000-b374-57eac9dced96/volumes" Oct 30 01:22:52.661931 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:52.661892 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:54.735204 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:54.735172 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.194007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.193964 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk"] Oct 30 01:22:56.194601 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.194300 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" podUID="784a87ff-53c7-4543-aac8-4c47e1b0e1f3" containerName="maas-api" containerID="cri-o://27478e6f03c15d30e7af9f8ebd200fa3c1dbbf5e99ea744a8e5d756335b74fb6" gracePeriod=30 Oct 30 01:22:56.259841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.259802 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft"] Oct 30 01:22:56.260091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.260061 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" containerID="cri-o://ff36c1db0433d17a419d7209c5227f0d37bd089d89506bd278ae046006299a9e" gracePeriod=30 Oct 30 01:22:56.281878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.281831 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.410386 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.410316 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.508676 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.508492 4517 generic.go:356] "Generic (PLEG): container finished" podID="784a87ff-53c7-4543-aac8-4c47e1b0e1f3" containerID="27478e6f03c15d30e7af9f8ebd200fa3c1dbbf5e99ea744a8e5d756335b74fb6" exitCode=0 Oct 30 01:22:56.509239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.509089 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" event={"ID":"784a87ff-53c7-4543-aac8-4c47e1b0e1f3","Type":"ContainerDied","Data":"27478e6f03c15d30e7af9f8ebd200fa3c1dbbf5e99ea744a8e5d756335b74fb6"} Oct 30 01:22:56.544755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.544717 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.596130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.596099 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:22:56.667826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.667686 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.726251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.726208 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-maas-api-tls\") pod \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " Oct 30 01:22:56.726484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.726382 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-metrics-certs\") pod \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " Oct 30 01:22:56.726484 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.726433 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-openshift-service-ca\") pod \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " Oct 30 01:22:56.726615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.726545 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-kube-api-access-6p5g6\" (UniqueName: \"kubernetes.io/projected/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-kube-api-access-6p5g6\") pod \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\" (UID: \"784a87ff-53c7-4543-aac8-4c47e1b0e1f3\") " Oct 30 01:22:56.727060 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.727026 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-openshift-service-ca" pod "784a87ff-53c7-4543-aac8-4c47e1b0e1f3" (UID: "784a87ff-53c7-4543-aac8-4c47e1b0e1f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:56.727489 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.727466 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.729050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.729017 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-kube-api-access-6p5g6" pod "784a87ff-53c7-4543-aac8-4c47e1b0e1f3" (UID: "784a87ff-53c7-4543-aac8-4c47e1b0e1f3"). InnerVolumeSpecName "kube-api-access-6p5g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:56.729189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.729066 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-metrics-certs" pod "784a87ff-53c7-4543-aac8-4c47e1b0e1f3" (UID: "784a87ff-53c7-4543-aac8-4c47e1b0e1f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:56.729189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.729100 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-maas-api-tls" pod "784a87ff-53c7-4543-aac8-4c47e1b0e1f3" (UID: "784a87ff-53c7-4543-aac8-4c47e1b0e1f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:56.838303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.837872 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5g6\" (UniqueName: \"kubernetes.io/projected/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-kube-api-access-6p5g6\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.838303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.837916 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.838303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.837931 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784a87ff-53c7-4543-aac8-4c47e1b0e1f3-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.988564 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:56.988511 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:57.516843 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:57.516810 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" Oct 30 01:22:57.517309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:57.516847 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" event={"ID":"784a87ff-53c7-4543-aac8-4c47e1b0e1f3","Type":"ContainerDied","Data":"9c3cc499ed9d89d587d6665fee261b6fa572c733939e079ce9225a8c75472b85"} Oct 30 01:22:57.517309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:57.516933 4517 scope.go:122] "RemoveContainer" containerID="27478e6f03c15d30e7af9f8ebd200fa3c1dbbf5e99ea744a8e5d756335b74fb6" Oct 30 01:22:57.576286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:57.576251 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:57.576286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:57.576280 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk"] Oct 30 01:22:57.586133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:57.586102 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk"] Oct 30 01:22:57.698335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:57.698295 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:58.099589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:58.099549 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:58.219320 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:58.219282 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:58.236619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:58.236582 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784a87ff-53c7-4543-aac8-4c47e1b0e1f3" path="/var/lib/kubelet/pods/784a87ff-53c7-4543-aac8-4c47e1b0e1f3/volumes" Oct 30 01:22:59.031194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:59.031153 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk"] Oct 30 01:22:59.041859 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:59.041823 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcdde58-276qk"] Oct 30 01:22:59.123768 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:59.123693 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:59.457483 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:22:59.457438 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-969b92-95fd57d7b-5jzhk" podUID="784a87ff-53c7-4543-aac8-4c47e1b0e1f3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.120:8443/health\": context deadline exceeded" Oct 30 01:23:00.236768 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:00.236696 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e267981-ea19-4f5f-be8c-9649fb997a6b" path="/var/lib/kubelet/pods/6e267981-ea19-4f5f-be8c-9649fb997a6b/volumes" Oct 30 01:23:00.762128 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:00.762084 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:23:01.264050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.264007 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:01.551144 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.551102 4517 generic.go:356] "Generic (PLEG): container finished" podID="1fa942ab-6c15-481f-8936-6b2c21358635" containerID="ff36c1db0433d17a419d7209c5227f0d37bd089d89506bd278ae046006299a9e" exitCode=0 Oct 30 01:23:01.551306 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.551164 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" event={"ID":"1fa942ab-6c15-481f-8936-6b2c21358635","Type":"ContainerDied","Data":"ff36c1db0433d17a419d7209c5227f0d37bd089d89506bd278ae046006299a9e"} Oct 30 01:23:01.551306 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.551218 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" event={"ID":"1fa942ab-6c15-481f-8936-6b2c21358635","Type":"ContainerDied","Data":"61e7f369ed3ad8b64ac95f5f51a172299a20b3f4a4fadd1b45add2888f9e2ce0"} Oct 30 01:23:01.551306 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.551236 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61e7f369ed3ad8b64ac95f5f51a172299a20b3f4a4fadd1b45add2888f9e2ce0" Oct 30 01:23:01.554350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.554334 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:23:01.645925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.645884 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-kube-api-access-97kp2\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-kube-api-access-97kp2\") pod \"1fa942ab-6c15-481f-8936-6b2c21358635\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " Oct 30 01:23:01.646152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.645949 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-certs\") pod \"1fa942ab-6c15-481f-8936-6b2c21358635\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " Oct 30 01:23:01.646152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646002 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-envoy\") pod \"1fa942ab-6c15-481f-8936-6b2c21358635\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " Oct 30 01:23:01.646152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646091 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fa942ab-6c15-481f-8936-6b2c21358635-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa942ab-6c15-481f-8936-6b2c21358635-istio-podinfo\") pod \"1fa942ab-6c15-481f-8936-6b2c21358635\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " Oct 30 01:23:01.646349 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646195 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-credential-socket\") pod \"1fa942ab-6c15-481f-8936-6b2c21358635\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " Oct 30 01:23:01.646349 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646244 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-istio-token\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-istio-token\") pod \"1fa942ab-6c15-481f-8936-6b2c21358635\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " Oct 30 01:23:01.646349 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646273 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa942ab-6c15-481f-8936-6b2c21358635-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa942ab-6c15-481f-8936-6b2c21358635-istiod-ca-cert\") pod \"1fa942ab-6c15-481f-8936-6b2c21358635\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " Oct 30 01:23:01.646349 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646322 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-socket\") pod \"1fa942ab-6c15-481f-8936-6b2c21358635\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " Oct 30 01:23:01.646551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646349 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-certs" pod "1fa942ab-6c15-481f-8936-6b2c21358635" (UID: "1fa942ab-6c15-481f-8936-6b2c21358635"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:01.646551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646405 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-data\") pod \"1fa942ab-6c15-481f-8936-6b2c21358635\" (UID: \"1fa942ab-6c15-481f-8936-6b2c21358635\") " Oct 30 01:23:01.646656 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646577 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-credential-socket" pod "1fa942ab-6c15-481f-8936-6b2c21358635" (UID: "1fa942ab-6c15-481f-8936-6b2c21358635"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:01.646836 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646809 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-socket" pod "1fa942ab-6c15-481f-8936-6b2c21358635" (UID: "1fa942ab-6c15-481f-8936-6b2c21358635"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:01.646906 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.646891 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-data" pod "1fa942ab-6c15-481f-8936-6b2c21358635" (UID: "1fa942ab-6c15-481f-8936-6b2c21358635"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:01.647088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.647056 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:01.647182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.647093 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:01.647182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.647106 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:01.647182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.647155 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:01.647299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.647189 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa942ab-6c15-481f-8936-6b2c21358635-istiod-ca-cert" pod "1fa942ab-6c15-481f-8936-6b2c21358635" (UID: "1fa942ab-6c15-481f-8936-6b2c21358635"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:01.649034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.648991 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-istio-token" pod "1fa942ab-6c15-481f-8936-6b2c21358635" (UID: "1fa942ab-6c15-481f-8936-6b2c21358635"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:01.649295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.649275 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fa942ab-6c15-481f-8936-6b2c21358635-istio-podinfo" pod "1fa942ab-6c15-481f-8936-6b2c21358635" (UID: "1fa942ab-6c15-481f-8936-6b2c21358635"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:01.649485 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.649468 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-kube-api-access-97kp2" pod "1fa942ab-6c15-481f-8936-6b2c21358635" (UID: "1fa942ab-6c15-481f-8936-6b2c21358635"). InnerVolumeSpecName "kube-api-access-97kp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:01.649670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.649652 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-envoy" pod "1fa942ab-6c15-481f-8936-6b2c21358635" (UID: "1fa942ab-6c15-481f-8936-6b2c21358635"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:01.748364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.748321 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97kp2\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-kube-api-access-97kp2\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:01.748364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.748362 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa942ab-6c15-481f-8936-6b2c21358635-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:01.748364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.748372 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa942ab-6c15-481f-8936-6b2c21358635-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:01.748646 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.748381 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa942ab-6c15-481f-8936-6b2c21358635-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:01.748646 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:01.748395 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa942ab-6c15-481f-8936-6b2c21358635-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:02.558243 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:02.558207 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft" Oct 30 01:23:02.617405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:02.617317 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft"] Oct 30 01:23:02.627774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:02.627742 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-h72ft"] Oct 30 01:23:03.493268 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:03.493231 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:03.682249 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:03.682198 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz"] Oct 30 01:23:03.683298 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:03.682455 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" containerID="cri-o://cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f" gracePeriod=30 Oct 30 01:23:04.235628 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:04.235590 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" path="/var/lib/kubelet/pods/1fa942ab-6c15-481f-8936-6b2c21358635/volumes" Oct 30 01:23:04.666359 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:04.666326 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:04.787278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:04.787231 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:04.939109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:04.939027 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:05.081491 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:05.081451 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:05.218945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:05.218844 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:05.358291 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:05.358253 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:06.441989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.441944 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp"] Oct 30 01:23:06.442563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442546 4517 cpu_manager.go:414] "containerMap: removing container" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" Oct 30 01:23:06.442563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442564 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442581 4517 cpu_manager.go:414] "containerMap: removing container" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442586 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442600 4517 cpu_manager.go:414] "containerMap: removing container" podUID="784a87ff-53c7-4543-aac8-4c47e1b0e1f3" containerName="maas-api" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442605 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784a87ff-53c7-4543-aac8-4c47e1b0e1f3" containerName="maas-api" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442614 4517 cpu_manager.go:414] "containerMap: removing container" podUID="2183c7f2-b9bf-4caa-9761-249a6736dc59" containerName="payload-processing" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442619 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2183c7f2-b9bf-4caa-9761-249a6736dc59" containerName="payload-processing" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442631 4517 cpu_manager.go:414] "containerMap: removing container" podUID="af096e76-cc77-421f-a309-e238ccc03f62" containerName="revoke-keys" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442636 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af096e76-cc77-421f-a309-e238ccc03f62" containerName="revoke-keys" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442648 4517 cpu_manager.go:414] "containerMap: removing container" podUID="73bd7b76-ad2a-47a6-8618-6d70ffd42ca1" containerName="payload-pre-processing" Oct 30 01:23:06.442685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442652 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bd7b76-ad2a-47a6-8618-6d70ffd42ca1" containerName="payload-pre-processing" Oct 30 01:23:06.443229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442745 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" Oct 30 01:23:06.443229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442755 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="784a87ff-53c7-4543-aac8-4c47e1b0e1f3" containerName="maas-api" Oct 30 01:23:06.443229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442762 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="2183c7f2-b9bf-4caa-9761-249a6736dc59" containerName="payload-processing" Oct 30 01:23:06.443229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442770 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" Oct 30 01:23:06.443229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442775 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="af096e76-cc77-421f-a309-e238ccc03f62" containerName="revoke-keys" Oct 30 01:23:06.443229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442781 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bd7b76-ad2a-47a6-8618-6d70ffd42ca1" containerName="payload-pre-processing" Oct 30 01:23:06.443229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.442792 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa942ab-6c15-481f-8936-6b2c21358635" containerName="istio-proxy" Oct 30 01:23:06.448017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.447984 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.472011 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.471976 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:23:06.472231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.472065 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:23:06.476658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.476611 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp"] Oct 30 01:23:06.538283 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.538242 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4gs\" (UniqueName: \"kubernetes.io/projected/beff6dd2-a384-43b8-a87c-1916420bf903-kube-api-access-nt4gs\") pod \"maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.538480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.538299 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.538480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.538462 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.640055 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.640011 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.640241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.640167 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4gs\" (UniqueName: \"kubernetes.io/projected/beff6dd2-a384-43b8-a87c-1916420bf903-kube-api-access-nt4gs\") pod \"maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.640241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.640196 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.640867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.640835 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.642932 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.642902 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.663436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.663399 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4gs\" (UniqueName: \"kubernetes.io/projected/beff6dd2-a384-43b8-a87c-1916420bf903-kube-api-access-nt4gs\") pod \"maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.764750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.764638 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:06.824618 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.824585 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:23:06.824831 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:06.824654 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:23:07.028231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:07.028146 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:07.107308 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:07.107268 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp"] Oct 30 01:23:07.112499 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:23:07.112115 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeff6dd2_a384_43b8_a87c_1916420bf903.slice/crio-77e0e52d4579c43fa100d0eece4a5991550d2b4b0560a21bd002ee88b540c8ab WatchSource:0}: Error finding container 77e0e52d4579c43fa100d0eece4a5991550d2b4b0560a21bd002ee88b540c8ab: Status 404 returned error can't find the container with id 77e0e52d4579c43fa100d0eece4a5991550d2b4b0560a21bd002ee88b540c8ab Oct 30 01:23:07.606323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:07.606285 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" event={"ID":"beff6dd2-a384-43b8-a87c-1916420bf903","Type":"ContainerStarted","Data":"dd6b71be78940dc81eb4f8eb1495f8bfb1d36d109ad132c945d76d9006defebb"} Oct 30 01:23:07.606918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:07.606358 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" event={"ID":"beff6dd2-a384-43b8-a87c-1916420bf903","Type":"ContainerStarted","Data":"77e0e52d4579c43fa100d0eece4a5991550d2b4b0560a21bd002ee88b540c8ab"} Oct 30 01:23:07.644385 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:07.644263 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" podStartSLOduration=1.644244222 podStartE2EDuration="1.644244222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:07.642934532 +0000 UTC m=+2925.555061674" watchObservedRunningTime="2026-10-30 01:23:07.644244222 +0000 UTC m=+2925.556371363" Oct 30 01:23:08.573506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:08.573467 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:08.629818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:08.629773 4517 generic.go:356] "Generic (PLEG): container finished" podID="beff6dd2-a384-43b8-a87c-1916420bf903" containerID="dd6b71be78940dc81eb4f8eb1495f8bfb1d36d109ad132c945d76d9006defebb" exitCode=0 Oct 30 01:23:08.630220 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:08.629855 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" event={"ID":"beff6dd2-a384-43b8-a87c-1916420bf903","Type":"ContainerDied","Data":"dd6b71be78940dc81eb4f8eb1495f8bfb1d36d109ad132c945d76d9006defebb"} Oct 30 01:23:09.107965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.107859 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:09.242309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.242266 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:09.254366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.254335 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:23:09.383523 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.383337 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:09.405588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.405556 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-socket\") pod \"3d1a870c-347f-4068-b738-c137c098d791\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " Oct 30 01:23:09.405788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.405654 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-kube-api-access-gbxrw\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-kube-api-access-gbxrw\") pod \"3d1a870c-347f-4068-b738-c137c098d791\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " Oct 30 01:23:09.405788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.405720 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-certs\") pod \"3d1a870c-347f-4068-b738-c137c098d791\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " Oct 30 01:23:09.405788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.405753 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-istio-token\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-istio-token\") pod \"3d1a870c-347f-4068-b738-c137c098d791\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " Oct 30 01:23:09.405788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.405780 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1a870c-347f-4068-b738-c137c098d791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1a870c-347f-4068-b738-c137c098d791-istiod-ca-cert\") pod \"3d1a870c-347f-4068-b738-c137c098d791\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " Oct 30 01:23:09.406034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.405842 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d1a870c-347f-4068-b738-c137c098d791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1a870c-347f-4068-b738-c137c098d791-istio-podinfo\") pod \"3d1a870c-347f-4068-b738-c137c098d791\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " Oct 30 01:23:09.406034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.405925 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-credential-socket\") pod \"3d1a870c-347f-4068-b738-c137c098d791\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " Oct 30 01:23:09.406034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.405986 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-data\") pod \"3d1a870c-347f-4068-b738-c137c098d791\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " Oct 30 01:23:09.406034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.405982 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-socket" pod "3d1a870c-347f-4068-b738-c137c098d791" (UID: "3d1a870c-347f-4068-b738-c137c098d791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.406255 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.406034 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-envoy\") pod \"3d1a870c-347f-4068-b738-c137c098d791\" (UID: \"3d1a870c-347f-4068-b738-c137c098d791\") " Oct 30 01:23:09.406255 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.406223 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-credential-socket" pod "3d1a870c-347f-4068-b738-c137c098d791" (UID: "3d1a870c-347f-4068-b738-c137c098d791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.406384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.406341 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-data" pod "3d1a870c-347f-4068-b738-c137c098d791" (UID: "3d1a870c-347f-4068-b738-c137c098d791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.406384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.406360 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1a870c-347f-4068-b738-c137c098d791-istiod-ca-cert" pod "3d1a870c-347f-4068-b738-c137c098d791" (UID: "3d1a870c-347f-4068-b738-c137c098d791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:09.406874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.406685 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-certs" pod "3d1a870c-347f-4068-b738-c137c098d791" (UID: "3d1a870c-347f-4068-b738-c137c098d791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.406874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.406719 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1a870c-347f-4068-b738-c137c098d791-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.406874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.406743 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.406874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.406759 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.406874 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.406772 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.408983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.408962 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-kube-api-access-gbxrw" pod "3d1a870c-347f-4068-b738-c137c098d791" (UID: "3d1a870c-347f-4068-b738-c137c098d791"). InnerVolumeSpecName "kube-api-access-gbxrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:09.409244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.409165 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d1a870c-347f-4068-b738-c137c098d791-istio-podinfo" pod "3d1a870c-347f-4068-b738-c137c098d791" (UID: "3d1a870c-347f-4068-b738-c137c098d791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:09.409421 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.409393 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-istio-token" pod "3d1a870c-347f-4068-b738-c137c098d791" (UID: "3d1a870c-347f-4068-b738-c137c098d791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:09.409839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.409820 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-envoy" pod "3d1a870c-347f-4068-b738-c137c098d791" (UID: "3d1a870c-347f-4068-b738-c137c098d791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.508045 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.507999 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1a870c-347f-4068-b738-c137c098d791-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.508045 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.508042 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.508374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.508057 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxrw\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-kube-api-access-gbxrw\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.508374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.508071 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1a870c-347f-4068-b738-c137c098d791-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.508374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.508084 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1a870c-347f-4068-b738-c137c098d791-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.638499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.638401 4517 generic.go:356] "Generic (PLEG): container finished" podID="3d1a870c-347f-4068-b738-c137c098d791" containerID="cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f" exitCode=0 Oct 30 01:23:09.638499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.638468 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" Oct 30 01:23:09.638499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.638488 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" event={"ID":"3d1a870c-347f-4068-b738-c137c098d791","Type":"ContainerDied","Data":"cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f"} Oct 30 01:23:09.639146 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.638558 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz" event={"ID":"3d1a870c-347f-4068-b738-c137c098d791","Type":"ContainerDied","Data":"b82cf0bfb9abb1776729e9004b0d9bb8d23fe04ad7d47163901d023a3e83ba4d"} Oct 30 01:23:09.639146 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.638592 4517 scope.go:122] "RemoveContainer" containerID="cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f" Oct 30 01:23:09.655573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.652964 4517 scope.go:122] "RemoveContainer" containerID="cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f" Oct 30 01:23:09.655573 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:09.654325 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f\": container with ID starting with cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f not found: ID does not exist" containerID="cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f" Oct 30 01:23:09.655573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.654364 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f"} err="failed to get container status \"cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f\": rpc error: code = NotFound desc = could not find container \"cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f\": container with ID starting with cf09eaa9a272c54e8b2ee4bc91ddea2b0e0a86122c448bc5e08083c57bd8127f not found: ID does not exist" Oct 30 01:23:09.687319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.687269 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz"] Oct 30 01:23:09.690946 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.690878 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-tb9fz"] Oct 30 01:23:09.743881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.743842 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:09.819589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.819551 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:09.913765 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.913651 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beff6dd2-a384-43b8-a87c-1916420bf903-kube-api-access-nt4gs\" (UniqueName: \"kubernetes.io/projected/beff6dd2-a384-43b8-a87c-1916420bf903-kube-api-access-nt4gs\") pod \"beff6dd2-a384-43b8-a87c-1916420bf903\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " Oct 30 01:23:09.913949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.913775 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-xks-ca\") pod \"beff6dd2-a384-43b8-a87c-1916420bf903\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " Oct 30 01:23:09.913993 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.913974 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-openshift-ca\") pod \"beff6dd2-a384-43b8-a87c-1916420bf903\" (UID: \"beff6dd2-a384-43b8-a87c-1916420bf903\") " Oct 30 01:23:09.914335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.914304 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-openshift-ca" pod "beff6dd2-a384-43b8-a87c-1916420bf903" (UID: "beff6dd2-a384-43b8-a87c-1916420bf903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:09.914558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.914432 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.916265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.916235 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-xks-ca" pod "beff6dd2-a384-43b8-a87c-1916420bf903" (UID: "beff6dd2-a384-43b8-a87c-1916420bf903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:09.916402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:09.916292 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beff6dd2-a384-43b8-a87c-1916420bf903-kube-api-access-nt4gs" pod "beff6dd2-a384-43b8-a87c-1916420bf903" (UID: "beff6dd2-a384-43b8-a87c-1916420bf903"). InnerVolumeSpecName "kube-api-access-nt4gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:10.016334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:10.016282 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt4gs\" (UniqueName: \"kubernetes.io/projected/beff6dd2-a384-43b8-a87c-1916420bf903-kube-api-access-nt4gs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.016334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:10.016325 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beff6dd2-a384-43b8-a87c-1916420bf903-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.237324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:10.237233 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1a870c-347f-4068-b738-c137c098d791" path="/var/lib/kubelet/pods/3d1a870c-347f-4068-b738-c137c098d791/volumes" Oct 30 01:23:10.647448 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:10.647416 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" Oct 30 01:23:10.647448 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:10.647430 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp" event={"ID":"beff6dd2-a384-43b8-a87c-1916420bf903","Type":"ContainerDied","Data":"77e0e52d4579c43fa100d0eece4a5991550d2b4b0560a21bd002ee88b540c8ab"} Oct 30 01:23:10.648016 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:10.647484 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77e0e52d4579c43fa100d0eece4a5991550d2b4b0560a21bd002ee88b540c8ab" Oct 30 01:23:10.880354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:10.880318 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:10.997873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:10.997772 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:12.071408 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:12.071369 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:14.230313 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:14.230260 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:15.982803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:15.982767 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:16.111668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:16.111632 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:16.325955 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:16.325918 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh"] Oct 30 01:23:16.326480 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:16.326449 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" containerID="cri-o://cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6" gracePeriod=30 Oct 30 01:23:16.367633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:16.367590 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:16.498732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:16.497421 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:16.631815 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:16.631695 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn"] Oct 30 01:23:16.773456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:16.773423 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:16.893624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:16.893524 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:17.197458 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:17.197376 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:17.511973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:17.511891 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:17.643736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:17.643688 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:17.738989 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:17.738954 4517 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:23:17.739171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:17.739021 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:23:18.168515 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:18.168468 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:18.809462 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:18.809427 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:18.926246 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:18.926210 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:19.185607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:19.185499 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:19.438199 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:19.438103 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:20.927172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:20.927113 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:21.632515 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.632471 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" podUID="32397089-ac22-4e3f-ba02-a642f2fb90c2" containerName="payload-processing" containerID="cri-o://fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592" gracePeriod=25 Oct 30 01:23:21.648577 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.648547 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 01:23:21.723940 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.723910 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-credential-socket\") pod \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " Oct 30 01:23:21.724145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724016 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-kube-api-access-zbvws\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-kube-api-access-zbvws\") pod \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " Oct 30 01:23:21.724145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724071 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-data\") pod \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " Oct 30 01:23:21.724145 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724114 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-podinfo\") pod \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " Oct 30 01:23:21.724354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724167 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-envoy\") pod \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " Oct 30 01:23:21.724354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724171 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-credential-socket" pod "8f5e313a-1534-49cc-bb1c-9ef6a29649e1" (UID: "8f5e313a-1534-49cc-bb1c-9ef6a29649e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:21.724354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724201 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istiod-ca-cert\") pod \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " Oct 30 01:23:21.724354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724222 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-socket\") pod \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " Oct 30 01:23:21.724354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724248 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-token\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-token\") pod \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " Oct 30 01:23:21.724354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724279 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-certs\") pod \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\" (UID: \"8f5e313a-1534-49cc-bb1c-9ef6a29649e1\") " Oct 30 01:23:21.724354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724315 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-data" pod "8f5e313a-1534-49cc-bb1c-9ef6a29649e1" (UID: "8f5e313a-1534-49cc-bb1c-9ef6a29649e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:21.724854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724578 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.724854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724588 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.724854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724760 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istiod-ca-cert" pod "8f5e313a-1534-49cc-bb1c-9ef6a29649e1" (UID: "8f5e313a-1534-49cc-bb1c-9ef6a29649e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:21.725025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724853 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-certs" pod "8f5e313a-1534-49cc-bb1c-9ef6a29649e1" (UID: "8f5e313a-1534-49cc-bb1c-9ef6a29649e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:21.725025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.724999 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-socket" pod "8f5e313a-1534-49cc-bb1c-9ef6a29649e1" (UID: "8f5e313a-1534-49cc-bb1c-9ef6a29649e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:21.727319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.727286 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-podinfo" pod "8f5e313a-1534-49cc-bb1c-9ef6a29649e1" (UID: "8f5e313a-1534-49cc-bb1c-9ef6a29649e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:21.727666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.727636 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-kube-api-access-zbvws" pod "8f5e313a-1534-49cc-bb1c-9ef6a29649e1" (UID: "8f5e313a-1534-49cc-bb1c-9ef6a29649e1"). InnerVolumeSpecName "kube-api-access-zbvws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:21.727834 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.727662 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-token" pod "8f5e313a-1534-49cc-bb1c-9ef6a29649e1" (UID: "8f5e313a-1534-49cc-bb1c-9ef6a29649e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:21.727912 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.727845 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-envoy" pod "8f5e313a-1534-49cc-bb1c-9ef6a29649e1" (UID: "8f5e313a-1534-49cc-bb1c-9ef6a29649e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:21.761892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.761854 4517 generic.go:356] "Generic (PLEG): container finished" podID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerID="cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6" exitCode=0 Oct 30 01:23:21.762085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.761920 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" Oct 30 01:23:21.762085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.761940 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" event={"ID":"8f5e313a-1534-49cc-bb1c-9ef6a29649e1","Type":"ContainerDied","Data":"cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6"} Oct 30 01:23:21.762085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.762009 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh" event={"ID":"8f5e313a-1534-49cc-bb1c-9ef6a29649e1","Type":"ContainerDied","Data":"032668312c15727e7de2b4181cd0bbd78dae50e280735db381eaaf8e297e16bc"} Oct 30 01:23:21.762085 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.762043 4517 scope.go:122] "RemoveContainer" containerID="cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6" Oct 30 01:23:21.777880 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.777847 4517 scope.go:122] "RemoveContainer" containerID="3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4" Oct 30 01:23:21.794448 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.794420 4517 scope.go:122] "RemoveContainer" containerID="cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6" Oct 30 01:23:21.794890 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:21.794859 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6\": container with ID starting with cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6 not found: ID does not exist" containerID="cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6" Oct 30 01:23:21.795021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.794904 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6"} err="failed to get container status \"cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6\": rpc error: code = NotFound desc = could not find container \"cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6\": container with ID starting with cc975b604577e8a7c48e8bb747b1931f0dbc9840b548ff81e4ea3e3382e2d9d6 not found: ID does not exist" Oct 30 01:23:21.795021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.794930 4517 scope.go:122] "RemoveContainer" containerID="3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4" Oct 30 01:23:21.795296 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:21.795271 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4\": container with ID starting with 3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4 not found: ID does not exist" containerID="3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4" Oct 30 01:23:21.795379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.795302 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4"} err="failed to get container status \"3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4\": rpc error: code = NotFound desc = could not find container \"3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4\": container with ID starting with 3d03e0dc08e54adb206e47ac1823f1dd93e9d21b6edc5e8216dfbf1b2e47fff4 not found: ID does not exist" Oct 30 01:23:21.822756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.822683 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh"] Oct 30 01:23:21.825654 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.825594 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.825909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.825888 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.825909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.825909 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbvws\" (UniqueName: \"kubernetes.io/projected/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-kube-api-access-zbvws\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.826080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.825918 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.826080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.825928 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.826080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.825939 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.826080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.825952 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5e313a-1534-49cc-bb1c-9ef6a29649e1-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.829333 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:21.829306 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-xlbkh"] Oct 30 01:23:22.235221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:22.235184 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" path="/var/lib/kubelet/pods/8f5e313a-1534-49cc-bb1c-9ef6a29649e1/volumes" Oct 30 01:23:22.539929 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:22.539896 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:23.918051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:23.918017 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:24.452909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:24.452873 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:24.571457 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:24.571425 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:24.693732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:24.693610 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:25.076193 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:25.076155 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:25.905854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:25.905814 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:26.031735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:26.031637 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:26.201798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:26.199797 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv"] Oct 30 01:23:26.201798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:26.200064 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" containerID="cri-o://d379c988c4115ebcf5531289c757b536d0b40e06e0c1a58264bad7cd4b6ab8b4" gracePeriod=30 Oct 30 01:23:26.239778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:26.239691 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:26.538452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:26.538413 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:26.723632 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:26.723592 4517 scope.go:122] "RemoveContainer" containerID="71e87236eb868f0a659e4ad77a86bd7ca9bc4e0e6e96050bcb52894dd080940c" Oct 30 01:23:26.734257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:26.734232 4517 scope.go:122] "RemoveContainer" containerID="9354ac7af7cb5b538e54fc9e09383daa76abaa03412f100ffbfba621947bfd95" Oct 30 01:23:26.744322 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:26.744298 4517 scope.go:122] "RemoveContainer" containerID="935f30db20bf788e127783881a091978b19bfc8cead2e7485a6d43be39cc7038" Oct 30 01:23:26.754295 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:26.754273 4517 scope.go:122] "RemoveContainer" containerID="ff36c1db0433d17a419d7209c5227f0d37bd089d89506bd278ae046006299a9e" Oct 30 01:23:27.037492 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:27.037456 4517 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:23:27.037672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:27.037537 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:23:28.350945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:28.350907 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:30.125758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:30.125661 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:30.761533 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:30.761493 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:23:31.096700 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.096657 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" podUID="32397089-ac22-4e3f-ba02-a642f2fb90c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:31.864887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.864846 4517 generic.go:356] "Generic (PLEG): container finished" podID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerID="d379c988c4115ebcf5531289c757b536d0b40e06e0c1a58264bad7cd4b6ab8b4" exitCode=0 Oct 30 01:23:31.865407 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.865075 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" event={"ID":"adb760a4-2540-4d84-b7d0-526f0b08985e","Type":"ContainerDied","Data":"d379c988c4115ebcf5531289c757b536d0b40e06e0c1a58264bad7cd4b6ab8b4"} Oct 30 01:23:31.918931 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.918902 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 01:23:31.948360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.948315 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:31.995475 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.995373 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-podinfo\") pod \"adb760a4-2540-4d84-b7d0-526f0b08985e\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " Oct 30 01:23:31.996364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.996327 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-token\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-token\") pod \"adb760a4-2540-4d84-b7d0-526f0b08985e\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " Oct 30 01:23:31.997013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.996995 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb760a4-2540-4d84-b7d0-526f0b08985e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb760a4-2540-4d84-b7d0-526f0b08985e-istiod-ca-cert\") pod \"adb760a4-2540-4d84-b7d0-526f0b08985e\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " Oct 30 01:23:31.997637 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.997512 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb760a4-2540-4d84-b7d0-526f0b08985e-istiod-ca-cert" pod "adb760a4-2540-4d84-b7d0-526f0b08985e" (UID: "adb760a4-2540-4d84-b7d0-526f0b08985e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:31.998148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.998129 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-kube-api-access-f59r6\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-kube-api-access-f59r6\") pod \"adb760a4-2540-4d84-b7d0-526f0b08985e\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " Oct 30 01:23:31.998370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.998352 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-credential-socket\") pod \"adb760a4-2540-4d84-b7d0-526f0b08985e\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " Oct 30 01:23:31.999324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:31.998799 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-credential-socket" pod "adb760a4-2540-4d84-b7d0-526f0b08985e" (UID: "adb760a4-2540-4d84-b7d0-526f0b08985e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:32.000055 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.000037 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-certs\") pod \"adb760a4-2540-4d84-b7d0-526f0b08985e\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " Oct 30 01:23:32.001399 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.000428 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-certs" pod "adb760a4-2540-4d84-b7d0-526f0b08985e" (UID: "adb760a4-2540-4d84-b7d0-526f0b08985e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:32.001813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.001747 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-data" pod "adb760a4-2540-4d84-b7d0-526f0b08985e" (UID: "adb760a4-2540-4d84-b7d0-526f0b08985e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:32.001968 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.001365 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-data\") pod \"adb760a4-2540-4d84-b7d0-526f0b08985e\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " Oct 30 01:23:32.002172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.002156 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-socket\") pod \"adb760a4-2540-4d84-b7d0-526f0b08985e\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " Oct 30 01:23:32.002575 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.002474 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-socket" pod "adb760a4-2540-4d84-b7d0-526f0b08985e" (UID: "adb760a4-2540-4d84-b7d0-526f0b08985e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:32.002854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.002836 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-envoy\") pod \"adb760a4-2540-4d84-b7d0-526f0b08985e\" (UID: \"adb760a4-2540-4d84-b7d0-526f0b08985e\") " Oct 30 01:23:32.003879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.003857 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.004020 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.004006 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.004129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.004116 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.004234 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.004221 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb760a4-2540-4d84-b7d0-526f0b08985e-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.004325 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.004314 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.010828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.010774 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-kube-api-access-f59r6" pod "adb760a4-2540-4d84-b7d0-526f0b08985e" (UID: "adb760a4-2540-4d84-b7d0-526f0b08985e"). InnerVolumeSpecName "kube-api-access-f59r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:32.012211 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.012183 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-podinfo" pod "adb760a4-2540-4d84-b7d0-526f0b08985e" (UID: "adb760a4-2540-4d84-b7d0-526f0b08985e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:32.012676 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.012655 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-envoy" pod "adb760a4-2540-4d84-b7d0-526f0b08985e" (UID: "adb760a4-2540-4d84-b7d0-526f0b08985e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:32.014265 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.014222 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-token" pod "adb760a4-2540-4d84-b7d0-526f0b08985e" (UID: "adb760a4-2540-4d84-b7d0-526f0b08985e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:32.097114 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.097063 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:32.105736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.105673 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.105910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.105748 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59r6\" (UniqueName: \"kubernetes.io/projected/adb760a4-2540-4d84-b7d0-526f0b08985e-kube-api-access-f59r6\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.105910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.105763 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.105910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.105776 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb760a4-2540-4d84-b7d0-526f0b08985e-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.398876 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.363244 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h"] Oct 30 01:23:32.398876 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.363523 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" containerID="cri-o://305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16" gracePeriod=30 Oct 30 01:23:32.398876 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.395784 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:32.590176 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.589530 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:32.739139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.739035 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:32.866456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.857967 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:32.903277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.903231 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" event={"ID":"adb760a4-2540-4d84-b7d0-526f0b08985e","Type":"ContainerDied","Data":"3647bbe414d5d546c8623cd200957dc807b4fff2ed366e783522b9ef0b4d5b83"} Oct 30 01:23:32.903476 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.903352 4517 scope.go:122] "RemoveContainer" containerID="d379c988c4115ebcf5531289c757b536d0b40e06e0c1a58264bad7cd4b6ab8b4" Oct 30 01:23:32.903549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.903534 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv" Oct 30 01:23:32.960262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.960228 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv"] Oct 30 01:23:32.971604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:32.971557 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-vwbjv"] Oct 30 01:23:33.251149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:33.251114 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:34.235987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:34.235949 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" path="/var/lib/kubelet/pods/adb760a4-2540-4d84-b7d0-526f0b08985e/volumes" Oct 30 01:23:34.612127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:34.612088 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:36.127015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:36.126958 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:23:36.127447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:36.127030 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:23:36.384143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:36.384057 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:37.689225 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.689195 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:23:37.713179 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713137 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-data\") pod \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " Oct 30 01:23:37.713387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713188 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istiod-ca-cert\") pod \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " Oct 30 01:23:37.713387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713231 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-socket\") pod \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " Oct 30 01:23:37.713387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713264 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-podinfo\") pod \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " Oct 30 01:23:37.713387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713324 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-kube-api-access-hrk4l\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-kube-api-access-hrk4l\") pod \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " Oct 30 01:23:37.713633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713388 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-certs\") pod \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " Oct 30 01:23:37.713633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713437 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-envoy\") pod \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " Oct 30 01:23:37.713633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713476 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-data" pod "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" (UID: "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:37.713633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713568 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-credential-socket\") pod \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " Oct 30 01:23:37.713633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713583 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-socket" pod "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" (UID: "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:37.713633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713609 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-token\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-token\") pod \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\" (UID: \"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de\") " Oct 30 01:23:37.714044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713682 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istiod-ca-cert" pod "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" (UID: "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:37.714044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.713913 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-certs" pod "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" (UID: "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:37.714157 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.714052 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-credential-socket" pod "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" (UID: "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:37.714457 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.714429 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:37.714520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.714456 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:37.714520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.714470 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:37.714520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.714483 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:37.714520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.714495 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:37.715956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.715923 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-podinfo" pod "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" (UID: "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:37.716073 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.715930 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-envoy" pod "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" (UID: "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:37.716369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.716350 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-token" pod "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" (UID: "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:37.716416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.716381 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-kube-api-access-hrk4l" pod "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" (UID: "8d3223b0-eab8-4a4f-ba99-ddd95c3da3de"). InnerVolumeSpecName "kube-api-access-hrk4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:37.815865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.815826 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:37.815865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.815859 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:37.815865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.815868 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrk4l\" (UniqueName: \"kubernetes.io/projected/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-kube-api-access-hrk4l\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:37.816133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.815877 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:37.937740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.937624 4517 generic.go:356] "Generic (PLEG): container finished" podID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerID="305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16" exitCode=0 Oct 30 01:23:37.937740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.937668 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" Oct 30 01:23:37.937740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.937685 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" event={"ID":"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de","Type":"ContainerDied","Data":"305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16"} Oct 30 01:23:37.938077 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.937753 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h" event={"ID":"8d3223b0-eab8-4a4f-ba99-ddd95c3da3de","Type":"ContainerDied","Data":"9c6b4cba8edbb588f5fbdbe415559ba6b49d6cc2f7353ac021b8774535b8c1f8"} Oct 30 01:23:37.938077 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.937784 4517 scope.go:122] "RemoveContainer" containerID="305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16" Oct 30 01:23:37.949456 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.949432 4517 scope.go:122] "RemoveContainer" containerID="305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16" Oct 30 01:23:37.949827 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:37.949808 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16\": container with ID starting with 305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16 not found: ID does not exist" containerID="305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16" Oct 30 01:23:37.949890 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.949839 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16"} err="failed to get container status \"305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16\": rpc error: code = NotFound desc = could not find container \"305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16\": container with ID starting with 305555f078d38c067b768df0d48cbd407cd48659b52f93290d46003ee6e33d16 not found: ID does not exist" Oct 30 01:23:37.990822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.990770 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h"] Oct 30 01:23:37.995104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:37.995075 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-xcq9h"] Oct 30 01:23:38.236104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:38.236024 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" path="/var/lib/kubelet/pods/8d3223b0-eab8-4a4f-ba99-ddd95c3da3de/volumes" Oct 30 01:23:38.739231 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:38.739187 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:39.795436 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:39.795400 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:39.920130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:39.918348 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:40.099821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:40.099722 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4"] Oct 30 01:23:40.099821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:40.099810 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj"] Oct 30 01:23:40.100103 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:40.100071 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" containerID="cri-o://1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef" gracePeriod=30 Oct 30 01:23:40.100553 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:40.100512 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" containerID="cri-o://1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a" gracePeriod=30 Oct 30 01:23:40.127678 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:40.127629 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:40.462191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:40.462104 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:40.621024 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:40.620986 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:41.097144 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.097109 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" podUID="32397089-ac22-4e3f-ba02-a642f2fb90c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:41.900136 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.900059 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:41.903303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.903277 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:23:41.971898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.971863 4517 generic.go:356] "Generic (PLEG): container finished" podID="32397089-ac22-4e3f-ba02-a642f2fb90c2" containerID="fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592" exitCode=0 Oct 30 01:23:41.972115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.971929 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" Oct 30 01:23:41.972115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.971946 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" event={"ID":"32397089-ac22-4e3f-ba02-a642f2fb90c2","Type":"ContainerDied","Data":"fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592"} Oct 30 01:23:41.972115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.972022 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn" event={"ID":"32397089-ac22-4e3f-ba02-a642f2fb90c2","Type":"ContainerDied","Data":"8429798f2c0f19c459445bda3cf57066431ff6506a34dc01ca8cad75fd29b371"} Oct 30 01:23:41.972115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.972056 4517 scope.go:122] "RemoveContainer" containerID="fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592" Oct 30 01:23:41.974591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.974562 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32397089-ac22-4e3f-ba02-a642f2fb90c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32397089-ac22-4e3f-ba02-a642f2fb90c2-plugins-config-volume\") pod \"32397089-ac22-4e3f-ba02-a642f2fb90c2\" (UID: \"32397089-ac22-4e3f-ba02-a642f2fb90c2\") " Oct 30 01:23:41.974871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.974838 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32397089-ac22-4e3f-ba02-a642f2fb90c2-kube-api-access-lnsld\" (UniqueName: \"kubernetes.io/projected/32397089-ac22-4e3f-ba02-a642f2fb90c2-kube-api-access-lnsld\") pod \"32397089-ac22-4e3f-ba02-a642f2fb90c2\" (UID: \"32397089-ac22-4e3f-ba02-a642f2fb90c2\") " Oct 30 01:23:41.974988 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.974916 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32397089-ac22-4e3f-ba02-a642f2fb90c2-plugins-config-volume" pod "32397089-ac22-4e3f-ba02-a642f2fb90c2" (UID: "32397089-ac22-4e3f-ba02-a642f2fb90c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:41.975551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:41.975526 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32397089-ac22-4e3f-ba02-a642f2fb90c2-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:42.023745 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:42.016993 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32397089-ac22-4e3f-ba02-a642f2fb90c2-kube-api-access-lnsld" pod "32397089-ac22-4e3f-ba02-a642f2fb90c2" (UID: "32397089-ac22-4e3f-ba02-a642f2fb90c2"). InnerVolumeSpecName "kube-api-access-lnsld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:42.062400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:42.062368 4517 scope.go:122] "RemoveContainer" containerID="fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592" Oct 30 01:23:42.062701 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:42.062678 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:42.063132 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:42.063105 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592\": container with ID starting with fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592 not found: ID does not exist" containerID="fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592" Oct 30 01:23:42.063258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:42.063144 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592"} err="failed to get container status \"fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592\": rpc error: code = NotFound desc = could not find container \"fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592\": container with ID starting with fc2ca202e4aaa835f48faba4278611512eaaebe196977bca0ce69355662ae592 not found: ID does not exist" Oct 30 01:23:42.077167 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:42.077127 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnsld\" (UniqueName: \"kubernetes.io/projected/32397089-ac22-4e3f-ba02-a642f2fb90c2-kube-api-access-lnsld\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:42.355200 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:42.355166 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn"] Oct 30 01:23:42.359493 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:42.359458 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae06fe-77c7cf7cfb-s4qwn"] Oct 30 01:23:43.093992 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:43.093960 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:43.430115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:43.430005 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:43.574445 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:43.574408 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:44.236775 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:44.236684 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32397089-ac22-4e3f-ba02-a642f2fb90c2" path="/var/lib/kubelet/pods/32397089-ac22-4e3f-ba02-a642f2fb90c2/volumes" Oct 30 01:23:44.761783 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:44.761748 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:45.416747 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.416667 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:23:45.456067 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.456034 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:23:45.542463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542427 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-istio-token\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-istio-token\") pod \"e6885190-f535-453a-bac8-1f067c2b6f78\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " Oct 30 01:23:45.542689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542488 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-envoy\") pod \"e41154ec-0a4b-4362-96c1-b381495cf243\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " Oct 30 01:23:45.542689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542525 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-istio-token\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-istio-token\") pod \"e41154ec-0a4b-4362-96c1-b381495cf243\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " Oct 30 01:23:45.542689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542557 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-credential-socket\") pod \"e41154ec-0a4b-4362-96c1-b381495cf243\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " Oct 30 01:23:45.542689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542606 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6885190-f535-453a-bac8-1f067c2b6f78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6885190-f535-453a-bac8-1f067c2b6f78-istiod-ca-cert\") pod \"e6885190-f535-453a-bac8-1f067c2b6f78\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " Oct 30 01:23:45.542689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542642 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-data\") pod \"e41154ec-0a4b-4362-96c1-b381495cf243\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " Oct 30 01:23:45.542689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542676 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-certs\") pod \"e6885190-f535-453a-bac8-1f067c2b6f78\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " Oct 30 01:23:45.543104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542751 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-credential-socket\") pod \"e6885190-f535-453a-bac8-1f067c2b6f78\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " Oct 30 01:23:45.543104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542782 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e41154ec-0a4b-4362-96c1-b381495cf243-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41154ec-0a4b-4362-96c1-b381495cf243-istio-podinfo\") pod \"e41154ec-0a4b-4362-96c1-b381495cf243\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " Oct 30 01:23:45.543104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542851 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-kube-api-access-ghrbq\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-kube-api-access-ghrbq\") pod \"e41154ec-0a4b-4362-96c1-b381495cf243\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " Oct 30 01:23:45.543104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542886 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-envoy\") pod \"e6885190-f535-453a-bac8-1f067c2b6f78\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " Oct 30 01:23:45.543104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542920 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-certs\") pod \"e41154ec-0a4b-4362-96c1-b381495cf243\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " Oct 30 01:23:45.543104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542941 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-credential-socket" pod "e41154ec-0a4b-4362-96c1-b381495cf243" (UID: "e41154ec-0a4b-4362-96c1-b381495cf243"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.543104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.542996 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41154ec-0a4b-4362-96c1-b381495cf243-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41154ec-0a4b-4362-96c1-b381495cf243-istiod-ca-cert\") pod \"e41154ec-0a4b-4362-96c1-b381495cf243\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " Oct 30 01:23:45.543104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543071 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-data\") pod \"e6885190-f535-453a-bac8-1f067c2b6f78\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " Oct 30 01:23:45.543698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543110 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6885190-f535-453a-bac8-1f067c2b6f78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6885190-f535-453a-bac8-1f067c2b6f78-istio-podinfo\") pod \"e6885190-f535-453a-bac8-1f067c2b6f78\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " Oct 30 01:23:45.543698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543135 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-socket\") pod \"e41154ec-0a4b-4362-96c1-b381495cf243\" (UID: \"e41154ec-0a4b-4362-96c1-b381495cf243\") " Oct 30 01:23:45.543698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543190 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-kube-api-access-tnqdg\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-kube-api-access-tnqdg\") pod \"e6885190-f535-453a-bac8-1f067c2b6f78\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " Oct 30 01:23:45.543698 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543222 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-socket\") pod \"e6885190-f535-453a-bac8-1f067c2b6f78\" (UID: \"e6885190-f535-453a-bac8-1f067c2b6f78\") " Oct 30 01:23:45.544540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543990 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.544540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543217 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-credential-socket" pod "e6885190-f535-453a-bac8-1f067c2b6f78" (UID: "e6885190-f535-453a-bac8-1f067c2b6f78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.544540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543491 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6885190-f535-453a-bac8-1f067c2b6f78-istiod-ca-cert" pod "e6885190-f535-453a-bac8-1f067c2b6f78" (UID: "e6885190-f535-453a-bac8-1f067c2b6f78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:45.544540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543744 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-data" pod "e41154ec-0a4b-4362-96c1-b381495cf243" (UID: "e41154ec-0a4b-4362-96c1-b381495cf243"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.544540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.543944 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-certs" pod "e6885190-f535-453a-bac8-1f067c2b6f78" (UID: "e6885190-f535-453a-bac8-1f067c2b6f78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.544540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.544116 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-data" pod "e6885190-f535-453a-bac8-1f067c2b6f78" (UID: "e6885190-f535-453a-bac8-1f067c2b6f78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.544540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.544230 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-certs" pod "e41154ec-0a4b-4362-96c1-b381495cf243" (UID: "e41154ec-0a4b-4362-96c1-b381495cf243"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.544540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.544432 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41154ec-0a4b-4362-96c1-b381495cf243-istiod-ca-cert" pod "e41154ec-0a4b-4362-96c1-b381495cf243" (UID: "e41154ec-0a4b-4362-96c1-b381495cf243"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:45.545830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.545690 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-envoy" pod "e41154ec-0a4b-4362-96c1-b381495cf243" (UID: "e41154ec-0a4b-4362-96c1-b381495cf243"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.546124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.546099 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-socket" pod "e6885190-f535-453a-bac8-1f067c2b6f78" (UID: "e6885190-f535-453a-bac8-1f067c2b6f78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.546226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.546119 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-istio-token" pod "e41154ec-0a4b-4362-96c1-b381495cf243" (UID: "e41154ec-0a4b-4362-96c1-b381495cf243"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:45.546331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.546308 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-socket" pod "e41154ec-0a4b-4362-96c1-b381495cf243" (UID: "e41154ec-0a4b-4362-96c1-b381495cf243"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.547776 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.547745 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-istio-token" pod "e6885190-f535-453a-bac8-1f067c2b6f78" (UID: "e6885190-f535-453a-bac8-1f067c2b6f78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:45.547887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.547866 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-kube-api-access-ghrbq" pod "e41154ec-0a4b-4362-96c1-b381495cf243" (UID: "e41154ec-0a4b-4362-96c1-b381495cf243"). InnerVolumeSpecName "kube-api-access-ghrbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:45.547966 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.547886 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6885190-f535-453a-bac8-1f067c2b6f78-istio-podinfo" pod "e6885190-f535-453a-bac8-1f067c2b6f78" (UID: "e6885190-f535-453a-bac8-1f067c2b6f78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:45.547966 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.547896 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-kube-api-access-tnqdg" pod "e6885190-f535-453a-bac8-1f067c2b6f78" (UID: "e6885190-f535-453a-bac8-1f067c2b6f78"). InnerVolumeSpecName "kube-api-access-tnqdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:45.548057 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.548020 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e41154ec-0a4b-4362-96c1-b381495cf243-istio-podinfo" pod "e41154ec-0a4b-4362-96c1-b381495cf243" (UID: "e41154ec-0a4b-4362-96c1-b381495cf243"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:45.548297 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.548267 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-envoy" pod "e6885190-f535-453a-bac8-1f067c2b6f78" (UID: "e6885190-f535-453a-bac8-1f067c2b6f78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.571721 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.571653 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct"] Oct 30 01:23:45.596282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.596240 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4"] Oct 30 01:23:45.645827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645732 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghrbq\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-kube-api-access-ghrbq\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.645827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645771 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.645827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645784 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.645827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645792 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41154ec-0a4b-4362-96c1-b381495cf243-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.645827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645801 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.645827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645809 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6885190-f535-453a-bac8-1f067c2b6f78-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.645827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645817 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.645827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645825 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnqdg\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-kube-api-access-tnqdg\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.645827 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645833 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.646259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645842 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6885190-f535-453a-bac8-1f067c2b6f78-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.646259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645850 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.646259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645857 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41154ec-0a4b-4362-96c1-b381495cf243-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.646259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645864 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6885190-f535-453a-bac8-1f067c2b6f78-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.646259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645871 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41154ec-0a4b-4362-96c1-b381495cf243-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.646259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645878 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.646259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645885 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6885190-f535-453a-bac8-1f067c2b6f78-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.646259 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.645892 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41154ec-0a4b-4362-96c1-b381495cf243-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.671262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:45.671221 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:46.006918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.006805 4517 generic.go:356] "Generic (PLEG): container finished" podID="e6885190-f535-453a-bac8-1f067c2b6f78" containerID="1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a" exitCode=0 Oct 30 01:23:46.006918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.006874 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" Oct 30 01:23:46.006918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.006898 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" event={"ID":"e6885190-f535-453a-bac8-1f067c2b6f78","Type":"ContainerDied","Data":"1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a"} Oct 30 01:23:46.007600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.006973 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4" event={"ID":"e6885190-f535-453a-bac8-1f067c2b6f78","Type":"ContainerDied","Data":"8cbdbeb978676f22de59b15238be5306c7991b1bb2918de5f2787acc5d4c8f2e"} Oct 30 01:23:46.007600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.007016 4517 scope.go:122] "RemoveContainer" containerID="1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a" Oct 30 01:23:46.010054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.010010 4517 generic.go:356] "Generic (PLEG): container finished" podID="e41154ec-0a4b-4362-96c1-b381495cf243" containerID="1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef" exitCode=0 Oct 30 01:23:46.010208 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.010118 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" event={"ID":"e41154ec-0a4b-4362-96c1-b381495cf243","Type":"ContainerDied","Data":"1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef"} Oct 30 01:23:46.010208 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.010177 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" event={"ID":"e41154ec-0a4b-4362-96c1-b381495cf243","Type":"ContainerDied","Data":"cfaa907b7d26499c3f5648097cbd38ce9cf42fac347e0a4ce9d92b6bba7311dc"} Oct 30 01:23:46.010339 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.010269 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj" Oct 30 01:23:46.023280 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.023257 4517 scope.go:122] "RemoveContainer" containerID="1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a" Oct 30 01:23:46.023746 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:46.023699 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a\": container with ID starting with 1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a not found: ID does not exist" containerID="1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a" Oct 30 01:23:46.023892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.023758 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a"} err="failed to get container status \"1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a\": rpc error: code = NotFound desc = could not find container \"1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a\": container with ID starting with 1ab2e7f2e629ea67388b68f6ce535beea5b5f1357a6c7716b8acf262ea22803a not found: ID does not exist" Oct 30 01:23:46.023892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.023793 4517 scope.go:122] "RemoveContainer" containerID="1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef" Oct 30 01:23:46.036621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.036590 4517 scope.go:122] "RemoveContainer" containerID="1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef" Oct 30 01:23:46.037084 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:46.037057 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef\": container with ID starting with 1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef not found: ID does not exist" containerID="1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef" Oct 30 01:23:46.037196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.037091 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef"} err="failed to get container status \"1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef\": rpc error: code = NotFound desc = could not find container \"1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef\": container with ID starting with 1a40b56c9b7e140cfa890b008fa14a6603d24f17adb52271e7735a8b432156ef not found: ID does not exist" Oct 30 01:23:46.064248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.064204 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4"] Oct 30 01:23:46.074589 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.074551 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dc6t4"] Oct 30 01:23:46.105437 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.105383 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj"] Oct 30 01:23:46.109400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.109361 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wllzj"] Oct 30 01:23:46.235727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.235672 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" path="/var/lib/kubelet/pods/e41154ec-0a4b-4362-96c1-b381495cf243/volumes" Oct 30 01:23:46.236286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.236268 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" path="/var/lib/kubelet/pods/e6885190-f535-453a-bac8-1f067c2b6f78/volumes" Oct 30 01:23:46.513027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:46.512992 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:48.065147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:48.065105 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:49.446373 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:49.446329 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:49.552672 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:49.552639 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72"] Oct 30 01:23:49.552947 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:49.552919 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" podUID="038d98e9-ef10-42e7-8bda-91dfe556550f" containerName="payload-pre-processing" containerID="cri-o://c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585" gracePeriod=30 Oct 30 01:23:49.624040 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:49.623985 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:49.736205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:49.736098 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:49.956513 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:49.956477 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:23:50.049935 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.049881 4517 generic.go:356] "Generic (PLEG): container finished" podID="038d98e9-ef10-42e7-8bda-91dfe556550f" containerID="c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585" exitCode=0 Oct 30 01:23:50.049935 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.049936 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" Oct 30 01:23:50.050260 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.049968 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" event={"ID":"038d98e9-ef10-42e7-8bda-91dfe556550f","Type":"ContainerDied","Data":"c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585"} Oct 30 01:23:50.050260 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.050033 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72" event={"ID":"038d98e9-ef10-42e7-8bda-91dfe556550f","Type":"ContainerDied","Data":"8d217d0387281e8513ca12ffb704c4f75241c5de133961e8ec2033908f85e1b1"} Oct 30 01:23:50.050260 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.050066 4517 scope.go:122] "RemoveContainer" containerID="c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585" Oct 30 01:23:50.061098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.061072 4517 scope.go:122] "RemoveContainer" containerID="c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585" Oct 30 01:23:50.061484 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:23:50.061460 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585\": container with ID starting with c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585 not found: ID does not exist" containerID="c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585" Oct 30 01:23:50.061544 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.061500 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585"} err="failed to get container status \"c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585\": rpc error: code = NotFound desc = could not find container \"c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585\": container with ID starting with c2f8d71139df1609439797aa1f512524602bbfd639be1e033da091a0022d1585 not found: ID does not exist" Oct 30 01:23:50.123119 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.123076 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038d98e9-ef10-42e7-8bda-91dfe556550f-kube-api-access-bczgt\" (UniqueName: \"kubernetes.io/projected/038d98e9-ef10-42e7-8bda-91dfe556550f-kube-api-access-bczgt\") pod \"038d98e9-ef10-42e7-8bda-91dfe556550f\" (UID: \"038d98e9-ef10-42e7-8bda-91dfe556550f\") " Oct 30 01:23:50.123410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.123381 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038d98e9-ef10-42e7-8bda-91dfe556550f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038d98e9-ef10-42e7-8bda-91dfe556550f-plugins-config-volume\") pod \"038d98e9-ef10-42e7-8bda-91dfe556550f\" (UID: \"038d98e9-ef10-42e7-8bda-91dfe556550f\") " Oct 30 01:23:50.124386 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.124350 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038d98e9-ef10-42e7-8bda-91dfe556550f-plugins-config-volume" pod "038d98e9-ef10-42e7-8bda-91dfe556550f" (UID: "038d98e9-ef10-42e7-8bda-91dfe556550f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:50.126459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.126436 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038d98e9-ef10-42e7-8bda-91dfe556550f-kube-api-access-bczgt" pod "038d98e9-ef10-42e7-8bda-91dfe556550f" (UID: "038d98e9-ef10-42e7-8bda-91dfe556550f"). InnerVolumeSpecName "kube-api-access-bczgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:50.185120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.185069 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:50.225343 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.225301 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bczgt\" (UniqueName: \"kubernetes.io/projected/038d98e9-ef10-42e7-8bda-91dfe556550f-kube-api-access-bczgt\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:50.225343 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.225341 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038d98e9-ef10-42e7-8bda-91dfe556550f-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:50.302593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.302501 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp"] Oct 30 01:23:50.307865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.307830 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abf8ae-m2lxp"] Oct 30 01:23:50.310841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.310809 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:50.402203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.402143 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72"] Oct 30 01:23:50.407560 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.407517 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abf8ae-7c4d75f895-9rs72"] Oct 30 01:23:50.426614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.426582 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:50.776576 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:50.776535 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:51.347234 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.347202 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:51.471444 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.471411 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.821582 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw"] Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823742 4517 cpu_manager.go:414] "containerMap: removing container" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823785 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823820 4517 cpu_manager.go:414] "containerMap: removing container" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823828 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823888 4517 cpu_manager.go:414] "containerMap: removing container" podUID="beff6dd2-a384-43b8-a87c-1916420bf903" containerName="revoke-keys" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823899 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beff6dd2-a384-43b8-a87c-1916420bf903" containerName="revoke-keys" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823918 4517 cpu_manager.go:414] "containerMap: removing container" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823934 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823945 4517 cpu_manager.go:414] "containerMap: removing container" podUID="038d98e9-ef10-42e7-8bda-91dfe556550f" containerName="payload-pre-processing" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823952 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038d98e9-ef10-42e7-8bda-91dfe556550f" containerName="payload-pre-processing" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823964 4517 cpu_manager.go:414] "containerMap: removing container" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823971 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823981 4517 cpu_manager.go:414] "containerMap: removing container" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.823988 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d5f4d6-77bb-4000-b374-57eac9dced96" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824066 4517 cpu_manager.go:414] "containerMap: removing container" podUID="32397089-ac22-4e3f-ba02-a642f2fb90c2" containerName="payload-processing" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824075 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32397089-ac22-4e3f-ba02-a642f2fb90c2" containerName="payload-processing" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824104 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824111 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824124 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824132 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824279 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="32397089-ac22-4e3f-ba02-a642f2fb90c2" containerName="payload-processing" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824293 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41154ec-0a4b-4362-96c1-b381495cf243" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824307 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6885190-f535-453a-bac8-1f067c2b6f78" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824318 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1a870c-347f-4068-b738-c137c098d791" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824330 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3223b0-eab8-4a4f-ba99-ddd95c3da3de" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824337 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="beff6dd2-a384-43b8-a87c-1916420bf903" containerName="revoke-keys" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824345 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824354 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="038d98e9-ef10-42e7-8bda-91dfe556550f" containerName="payload-pre-processing" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824363 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb760a4-2540-4d84-b7d0-526f0b08985e" containerName="istio-proxy" Oct 30 01:23:51.826742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.824372 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" Oct 30 01:23:51.852737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.833084 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:51.903614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.903576 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw"] Oct 30 01:23:51.908311 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.908264 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:23:51.908564 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.908334 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:23:51.955177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.955137 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5edaaa3d-rddcw\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:51.955353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.955208 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zqf7\" (UniqueName: \"kubernetes.io/projected/f56f3a12-1267-487b-b597-ec9514c6a26d-kube-api-access-5zqf7\") pod \"maas-api-revoke-sub-5edaaa3d-rddcw\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:51.955353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:51.955290 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5edaaa3d-rddcw\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:52.056964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.056922 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zqf7\" (UniqueName: \"kubernetes.io/projected/f56f3a12-1267-487b-b597-ec9514c6a26d-kube-api-access-5zqf7\") pod \"maas-api-revoke-sub-5edaaa3d-rddcw\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:52.057162 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.056973 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5edaaa3d-rddcw\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:52.057162 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.057091 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5edaaa3d-rddcw\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:52.057831 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.057797 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5edaaa3d-rddcw\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:52.059881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.059849 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5edaaa3d-rddcw\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:52.080505 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.080419 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zqf7\" (UniqueName: \"kubernetes.io/projected/f56f3a12-1267-487b-b597-ec9514c6a26d-kube-api-access-5zqf7\") pod \"maas-api-revoke-sub-5edaaa3d-rddcw\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:52.221640 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.221604 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:52.235354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.235317 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038d98e9-ef10-42e7-8bda-91dfe556550f" path="/var/lib/kubelet/pods/038d98e9-ef10-42e7-8bda-91dfe556550f/volumes" Oct 30 01:23:52.235796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.235781 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beff6dd2-a384-43b8-a87c-1916420bf903" path="/var/lib/kubelet/pods/beff6dd2-a384-43b8-a87c-1916420bf903/volumes" Oct 30 01:23:52.408455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.408360 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:52.528263 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:52.528233 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw"] Oct 30 01:23:52.531627 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:23:52.531592 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56f3a12_1267_487b_b597_ec9514c6a26d.slice/crio-571b2b51344d387f24878ba5d99515dfee4ca39aed56abf4c8317eaef9c9903e WatchSource:0}: Error finding container 571b2b51344d387f24878ba5d99515dfee4ca39aed56abf4c8317eaef9c9903e: Status 404 returned error can't find the container with id 571b2b51344d387f24878ba5d99515dfee4ca39aed56abf4c8317eaef9c9903e Oct 30 01:23:53.090587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:53.090542 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" event={"ID":"f56f3a12-1267-487b-b597-ec9514c6a26d","Type":"ContainerStarted","Data":"b2b7e92bf98bf5cf6428840f40472683eb91ca6946e22b7afc645e5c0838e14f"} Oct 30 01:23:53.091175 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:53.090614 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" event={"ID":"f56f3a12-1267-487b-b597-ec9514c6a26d","Type":"ContainerStarted","Data":"571b2b51344d387f24878ba5d99515dfee4ca39aed56abf4c8317eaef9c9903e"} Oct 30 01:23:53.131074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:53.130986 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" podStartSLOduration=2.1309605879999998 podStartE2EDuration="2.130960588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:53.129663054 +0000 UTC m=+2971.041790195" watchObservedRunningTime="2026-10-30 01:23:53.130960588 +0000 UTC m=+2971.043087729" Oct 30 01:23:54.098988 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:54.098954 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:54.099591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:54.099565 4517 generic.go:356] "Generic (PLEG): container finished" podID="f56f3a12-1267-487b-b597-ec9514c6a26d" containerID="b2b7e92bf98bf5cf6428840f40472683eb91ca6946e22b7afc645e5c0838e14f" exitCode=0 Oct 30 01:23:54.099690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:54.099664 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" event={"ID":"f56f3a12-1267-487b-b597-ec9514c6a26d","Type":"ContainerDied","Data":"b2b7e92bf98bf5cf6428840f40472683eb91ca6946e22b7afc645e5c0838e14f"} Oct 30 01:23:55.271762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.271722 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:55.409531 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.409436 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-xks-ca\") pod \"f56f3a12-1267-487b-b597-ec9514c6a26d\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " Oct 30 01:23:55.409531 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.409478 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56f3a12-1267-487b-b597-ec9514c6a26d-kube-api-access-5zqf7\" (UniqueName: \"kubernetes.io/projected/f56f3a12-1267-487b-b597-ec9514c6a26d-kube-api-access-5zqf7\") pod \"f56f3a12-1267-487b-b597-ec9514c6a26d\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " Oct 30 01:23:55.409821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.409590 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-openshift-ca\") pod \"f56f3a12-1267-487b-b597-ec9514c6a26d\" (UID: \"f56f3a12-1267-487b-b597-ec9514c6a26d\") " Oct 30 01:23:55.410049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.410018 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-openshift-ca" pod "f56f3a12-1267-487b-b597-ec9514c6a26d" (UID: "f56f3a12-1267-487b-b597-ec9514c6a26d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:55.411901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.411875 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-xks-ca" pod "f56f3a12-1267-487b-b597-ec9514c6a26d" (UID: "f56f3a12-1267-487b-b597-ec9514c6a26d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:55.412005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.411970 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56f3a12-1267-487b-b597-ec9514c6a26d-kube-api-access-5zqf7" pod "f56f3a12-1267-487b-b597-ec9514c6a26d" (UID: "f56f3a12-1267-487b-b597-ec9514c6a26d"). InnerVolumeSpecName "kube-api-access-5zqf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:55.510695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.510661 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.510695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.510694 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56f3a12-1267-487b-b597-ec9514c6a26d-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.510695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.510716 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zqf7\" (UniqueName: \"kubernetes.io/projected/f56f3a12-1267-487b-b597-ec9514c6a26d-kube-api-access-5zqf7\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.832478 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.832436 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:55.972159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:55.972105 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:56.092500 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:56.092399 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:56.115036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:56.115004 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" Oct 30 01:23:56.115036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:56.115024 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw" event={"ID":"f56f3a12-1267-487b-b597-ec9514c6a26d","Type":"ContainerDied","Data":"571b2b51344d387f24878ba5d99515dfee4ca39aed56abf4c8317eaef9c9903e"} Oct 30 01:23:56.115284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:56.115111 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="571b2b51344d387f24878ba5d99515dfee4ca39aed56abf4c8317eaef9c9903e" Oct 30 01:23:56.349981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:56.349901 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:57.835659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:57.835598 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:58.328893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:58.328860 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:58.625735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:58.625615 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:58.765762 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:58.765681 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:59.184143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:23:59.184108 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:00.600409 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.600324 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" containerID="cri-o://252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838" gracePeriod=45 Oct 30 01:24:00.621419 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.621361 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" containerID="cri-o://3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34" gracePeriod=45 Oct 30 01:24:00.761879 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:24:00.761794 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:24:00.939774 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.939700 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 01:24:00.943541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.943512 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 01:24:00.994490 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994448 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-home\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-home\") pod \"ec3f002e-af8a-40f2-a644-59662eaff603\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " Oct 30 01:24:00.994692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994503 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec3f002e-af8a-40f2-a644-59662eaff603-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec3f002e-af8a-40f2-a644-59662eaff603-tls-certs\") pod \"ec3f002e-af8a-40f2-a644-59662eaff603\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " Oct 30 01:24:00.994692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994552 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tls-certs\" (UniqueName: \"kubernetes.io/secret/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tls-certs\") pod \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " Oct 30 01:24:00.994692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994631 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-home\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-home\") pod \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " Oct 30 01:24:00.994692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994687 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-model-cache\") pod \"ec3f002e-af8a-40f2-a644-59662eaff603\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " Oct 30 01:24:00.994978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994804 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-dshm\") pod \"ec3f002e-af8a-40f2-a644-59662eaff603\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " Oct 30 01:24:00.994978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994857 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3f002e-af8a-40f2-a644-59662eaff603-kube-api-access-zwhd2\" (UniqueName: \"kubernetes.io/projected/ec3f002e-af8a-40f2-a644-59662eaff603-kube-api-access-zwhd2\") pod \"ec3f002e-af8a-40f2-a644-59662eaff603\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " Oct 30 01:24:00.994978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994889 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tmp-dir\") pod \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " Oct 30 01:24:00.994978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994916 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-tmp-dir\") pod \"ec3f002e-af8a-40f2-a644-59662eaff603\" (UID: \"ec3f002e-af8a-40f2-a644-59662eaff603\") " Oct 30 01:24:00.994978 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.994956 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-dshm\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-dshm\") pod \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " Oct 30 01:24:00.995424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.995007 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-kube-api-access-xv4tq\" (UniqueName: \"kubernetes.io/projected/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-kube-api-access-xv4tq\") pod \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " Oct 30 01:24:00.995424 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.995055 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-model-cache\") pod \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\" (UID: \"340bbdbe-47e6-43d3-b384-e2f5a8e4b726\") " Oct 30 01:24:00.996271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.996241 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-tmp-dir" pod "ec3f002e-af8a-40f2-a644-59662eaff603" (UID: "ec3f002e-af8a-40f2-a644-59662eaff603"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.996662 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.996633 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tmp-dir" pod "340bbdbe-47e6-43d3-b384-e2f5a8e4b726" (UID: "340bbdbe-47e6-43d3-b384-e2f5a8e4b726"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.996842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.996657 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-home" pod "340bbdbe-47e6-43d3-b384-e2f5a8e4b726" (UID: "340bbdbe-47e6-43d3-b384-e2f5a8e4b726"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.997009 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.996987 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-home" pod "ec3f002e-af8a-40f2-a644-59662eaff603" (UID: "ec3f002e-af8a-40f2-a644-59662eaff603"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.997258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.997225 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-model-cache" pod "ec3f002e-af8a-40f2-a644-59662eaff603" (UID: "ec3f002e-af8a-40f2-a644-59662eaff603"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.997666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:00.997634 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-model-cache" pod "340bbdbe-47e6-43d3-b384-e2f5a8e4b726" (UID: "340bbdbe-47e6-43d3-b384-e2f5a8e4b726"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:01.004018 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.003971 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-dshm" pod "340bbdbe-47e6-43d3-b384-e2f5a8e4b726" (UID: "340bbdbe-47e6-43d3-b384-e2f5a8e4b726"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:01.004125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.004012 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tls-certs" pod "340bbdbe-47e6-43d3-b384-e2f5a8e4b726" (UID: "340bbdbe-47e6-43d3-b384-e2f5a8e4b726"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:01.004189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.004147 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-dshm" pod "ec3f002e-af8a-40f2-a644-59662eaff603" (UID: "ec3f002e-af8a-40f2-a644-59662eaff603"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:01.004189 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.004147 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3f002e-af8a-40f2-a644-59662eaff603-kube-api-access-zwhd2" pod "ec3f002e-af8a-40f2-a644-59662eaff603" (UID: "ec3f002e-af8a-40f2-a644-59662eaff603"). InnerVolumeSpecName "kube-api-access-zwhd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:01.005667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.005633 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec3f002e-af8a-40f2-a644-59662eaff603-tls-certs" pod "ec3f002e-af8a-40f2-a644-59662eaff603" (UID: "ec3f002e-af8a-40f2-a644-59662eaff603"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:01.006375 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.006345 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-kube-api-access-xv4tq" pod "340bbdbe-47e6-43d3-b384-e2f5a8e4b726" (UID: "340bbdbe-47e6-43d3-b384-e2f5a8e4b726"). InnerVolumeSpecName "kube-api-access-xv4tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:01.014053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.014025 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:01.096766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096689 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwhd2\" (UniqueName: \"kubernetes.io/projected/ec3f002e-af8a-40f2-a644-59662eaff603-kube-api-access-zwhd2\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096786 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096805 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096814 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096823 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv4tq\" (UniqueName: \"kubernetes.io/projected/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-kube-api-access-xv4tq\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096833 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096841 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096850 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec3f002e-af8a-40f2-a644-59662eaff603-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096857 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096864 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340bbdbe-47e6-43d3-b384-e2f5a8e4b726-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096877 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.097004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.096887 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec3f002e-af8a-40f2-a644-59662eaff603-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:01.189882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.189771 4517 generic.go:356] "Generic (PLEG): container finished" podID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerID="3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34" exitCode=0 Oct 30 01:24:01.189882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.189839 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" Oct 30 01:24:01.190165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.189872 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" event={"ID":"340bbdbe-47e6-43d3-b384-e2f5a8e4b726","Type":"ContainerDied","Data":"3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34"} Oct 30 01:24:01.190165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.189928 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4" event={"ID":"340bbdbe-47e6-43d3-b384-e2f5a8e4b726","Type":"ContainerDied","Data":"933b877a632f5fbb3c9585664917e385befd1b271705d01626b87226d70026a8"} Oct 30 01:24:01.190165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.189961 4517 scope.go:122] "RemoveContainer" containerID="3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34" Oct 30 01:24:01.191685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.191653 4517 generic.go:356] "Generic (PLEG): container finished" podID="ec3f002e-af8a-40f2-a644-59662eaff603" containerID="252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838" exitCode=0 Oct 30 01:24:01.192071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.191702 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" Oct 30 01:24:01.192071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.191749 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" event={"ID":"ec3f002e-af8a-40f2-a644-59662eaff603","Type":"ContainerDied","Data":"252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838"} Oct 30 01:24:01.192071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.191788 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct" event={"ID":"ec3f002e-af8a-40f2-a644-59662eaff603","Type":"ContainerDied","Data":"f65493fb99080a42b17930aa099b877c5b1861ece5c916dc707ac4621fe28893"} Oct 30 01:24:01.204732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.204685 4517 scope.go:122] "RemoveContainer" containerID="3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34" Oct 30 01:24:01.205162 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:24:01.205140 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34\": container with ID starting with 3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34 not found: ID does not exist" containerID="3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34" Oct 30 01:24:01.205251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.205173 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34"} err="failed to get container status \"3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34\": rpc error: code = NotFound desc = could not find container \"3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34\": container with ID starting with 3227b6ba1c7c44e1a0a58cd2a470cfe0096ed8b558656b8f382ebc9c60d63a34 not found: ID does not exist" Oct 30 01:24:01.205251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.205192 4517 scope.go:122] "RemoveContainer" containerID="252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838" Oct 30 01:24:01.216385 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.216355 4517 scope.go:122] "RemoveContainer" containerID="252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838" Oct 30 01:24:01.216824 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:24:01.216801 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838\": container with ID starting with 252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838 not found: ID does not exist" containerID="252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838" Oct 30 01:24:01.216910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.216833 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838"} err="failed to get container status \"252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838\": rpc error: code = NotFound desc = could not find container \"252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838\": container with ID starting with 252212c793da52c2cfb0022ddcdc11068f1d44f3308590b42e602b984072a838 not found: ID does not exist" Oct 30 01:24:01.249651 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.249610 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4"] Oct 30 01:24:01.254789 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.254758 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/simulator-w2-40ac41-kserve-fb5b597d4-z58z4"] Oct 30 01:24:01.286494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.286445 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct"] Oct 30 01:24:01.290982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:01.290946 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-2-w2-40ac41-kserve-6bdbcd9c67-tpdct"] Oct 30 01:24:02.235093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:02.235056 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" path="/var/lib/kubelet/pods/340bbdbe-47e6-43d3-b384-e2f5a8e4b726/volumes" Oct 30 01:24:02.235597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:02.235580 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" path="/var/lib/kubelet/pods/ec3f002e-af8a-40f2-a644-59662eaff603/volumes" Oct 30 01:24:02.941722 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:02.941674 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:05.064370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:05.064328 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:05.190501 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:05.190458 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:05.303757 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:05.303678 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:05.508202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:05.508102 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v"] Oct 30 01:24:05.508394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:05.508369 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" containerID="cri-o://b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273" gracePeriod=30 Oct 30 01:24:05.522246 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:05.522214 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:05.988733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:05.988644 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:06.879559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:06.879516 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:08.834054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:08.834023 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:10.532100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.532071 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:10.811634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.811608 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 01:24:10.920125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920089 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-envoy\") pod \"35e1387e-0825-4d7d-b5c5-e1282d033a11\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " Oct 30 01:24:10.920331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920144 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-certs\") pod \"35e1387e-0825-4d7d-b5c5-e1282d033a11\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " Oct 30 01:24:10.920331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920188 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-socket\") pod \"35e1387e-0825-4d7d-b5c5-e1282d033a11\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " Oct 30 01:24:10.920331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920282 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e1387e-0825-4d7d-b5c5-e1282d033a11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e1387e-0825-4d7d-b5c5-e1282d033a11-istiod-ca-cert\") pod \"35e1387e-0825-4d7d-b5c5-e1282d033a11\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " Oct 30 01:24:10.920331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920312 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-data\") pod \"35e1387e-0825-4d7d-b5c5-e1282d033a11\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " Oct 30 01:24:10.920572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920376 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-podinfo\") pod \"35e1387e-0825-4d7d-b5c5-e1282d033a11\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " Oct 30 01:24:10.920572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920454 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-kube-api-access-9m569\") pod \"35e1387e-0825-4d7d-b5c5-e1282d033a11\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " Oct 30 01:24:10.920572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920490 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-certs" pod "35e1387e-0825-4d7d-b5c5-e1282d033a11" (UID: "35e1387e-0825-4d7d-b5c5-e1282d033a11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:10.920572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920533 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-token\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-token\") pod \"35e1387e-0825-4d7d-b5c5-e1282d033a11\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " Oct 30 01:24:10.920816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920571 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-credential-socket\") pod \"35e1387e-0825-4d7d-b5c5-e1282d033a11\" (UID: \"35e1387e-0825-4d7d-b5c5-e1282d033a11\") " Oct 30 01:24:10.920816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920578 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-socket" pod "35e1387e-0825-4d7d-b5c5-e1282d033a11" (UID: "35e1387e-0825-4d7d-b5c5-e1282d033a11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:10.920816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920652 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-data" pod "35e1387e-0825-4d7d-b5c5-e1282d033a11" (UID: "35e1387e-0825-4d7d-b5c5-e1282d033a11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:10.920816 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920670 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e1387e-0825-4d7d-b5c5-e1282d033a11-istiod-ca-cert" pod "35e1387e-0825-4d7d-b5c5-e1282d033a11" (UID: "35e1387e-0825-4d7d-b5c5-e1282d033a11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:10.921002 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.920880 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-credential-socket" pod "35e1387e-0825-4d7d-b5c5-e1282d033a11" (UID: "35e1387e-0825-4d7d-b5c5-e1282d033a11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:10.921416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.921156 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:10.921416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.921183 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:10.921416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.921270 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:10.921416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.921375 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e1387e-0825-4d7d-b5c5-e1282d033a11-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:10.921416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.921389 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:10.922750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.922691 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-podinfo" pod "35e1387e-0825-4d7d-b5c5-e1282d033a11" (UID: "35e1387e-0825-4d7d-b5c5-e1282d033a11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:10.923134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.923112 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-token" pod "35e1387e-0825-4d7d-b5c5-e1282d033a11" (UID: "35e1387e-0825-4d7d-b5c5-e1282d033a11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:10.923273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.923253 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-envoy" pod "35e1387e-0825-4d7d-b5c5-e1282d033a11" (UID: "35e1387e-0825-4d7d-b5c5-e1282d033a11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:10.923369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:10.923289 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-kube-api-access-9m569" pod "35e1387e-0825-4d7d-b5c5-e1282d033a11" (UID: "35e1387e-0825-4d7d-b5c5-e1282d033a11"). InnerVolumeSpecName "kube-api-access-9m569". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:11.022287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.022243 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.022287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.022278 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.022287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.022287 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e1387e-0825-4d7d-b5c5-e1282d033a11-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.022287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.022297 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/35e1387e-0825-4d7d-b5c5-e1282d033a11-kube-api-access-9m569\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.044111 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.044008 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:11.264194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.264137 4517 generic.go:356] "Generic (PLEG): container finished" podID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerID="b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273" exitCode=0 Oct 30 01:24:11.264533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.264209 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" Oct 30 01:24:11.264533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.264218 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" event={"ID":"35e1387e-0825-4d7d-b5c5-e1282d033a11","Type":"ContainerDied","Data":"b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273"} Oct 30 01:24:11.264533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.264360 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v" event={"ID":"35e1387e-0825-4d7d-b5c5-e1282d033a11","Type":"ContainerDied","Data":"06cf1737591ff90d86b286af8ac46a3e5544480db3bf109acd34cd851ebeece2"} Oct 30 01:24:11.264533 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.264400 4517 scope.go:122] "RemoveContainer" containerID="b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273" Oct 30 01:24:11.278143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.278116 4517 scope.go:122] "RemoveContainer" containerID="d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621" Oct 30 01:24:11.291615 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.291588 4517 scope.go:122] "RemoveContainer" containerID="b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273" Oct 30 01:24:11.292016 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:24:11.291991 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273\": container with ID starting with b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273 not found: ID does not exist" containerID="b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273" Oct 30 01:24:11.292148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.292033 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273"} err="failed to get container status \"b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273\": rpc error: code = NotFound desc = could not find container \"b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273\": container with ID starting with b809794f5971dc42b35e9636a71501de5b643ff26f60274a4e4fc01e961a1273 not found: ID does not exist" Oct 30 01:24:11.292148 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.292061 4517 scope.go:122] "RemoveContainer" containerID="d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621" Oct 30 01:24:11.292445 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:24:11.292420 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621\": container with ID starting with d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621 not found: ID does not exist" containerID="d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621" Oct 30 01:24:11.292579 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.292454 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621"} err="failed to get container status \"d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621\": rpc error: code = NotFound desc = could not find container \"d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621\": container with ID starting with d19d432cc3845a028904d241aba8d6ae02a5d827f13c948bbda9766722dbe621 not found: ID does not exist" Oct 30 01:24:11.314835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.314734 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v"] Oct 30 01:24:11.324809 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:11.324757 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-djc7v"] Oct 30 01:24:12.151513 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:12.151469 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:12.237063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:12.237029 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" path="/var/lib/kubelet/pods/35e1387e-0825-4d7d-b5c5-e1282d033a11/volumes" Oct 30 01:24:13.746012 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:13.745947 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:15.119447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:15.119398 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:15.291377 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:15.291339 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:15.408500 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:15.408404 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:15.726050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:15.725962 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:15.842616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:15.842572 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:15.957910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:15.957872 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:16.330788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:16.330742 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:16.713183 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:16.713069 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:16.835957 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:16.835918 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:17.403200 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:17.403153 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:17.910730 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:17.910671 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:18.092948 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:18.092905 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:18.203864 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:18.203764 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:18.536122 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:18.536086 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:19.290123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:19.290093 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:20.558646 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:20.558604 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:22.285473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:22.285435 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:22.526402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:22.526338 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:22.674581 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:22.674460 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:23.850997 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:23.850957 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:24.000445 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:24.000401 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:24.235787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:24.235662 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m"] Oct 30 01:24:24.235971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:24.235908 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" containerID="cri-o://2663e73863508e507556d1ccadf38601f3a46c545ca15add3e9a4b3220d715f1" gracePeriod=30 Oct 30 01:24:24.274368 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:24.274324 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:24.447590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:24.447548 4517 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:24:24.447790 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:24.447616 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:24:25.009044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:25.009008 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:25.186060 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:25.186017 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:26.289535 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:26.289497 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:26.427634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:26.427593 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:26.566409 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:26.566300 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4"] Oct 30 01:24:26.567088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:26.567051 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" containerID="cri-o://eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5" gracePeriod=30 Oct 30 01:24:26.600922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:26.600882 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:26.847303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:26.847219 4517 scope.go:122] "RemoveContainer" containerID="f273cd7a96c7bae797b16d24e081868700fff91115af9e1585341b63ab7b040e" Oct 30 01:24:26.857570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:26.857545 4517 scope.go:122] "RemoveContainer" containerID="ad70b0e37a8acf68d15af933f773e8a50e1f7498ae1f70284d58738d46ef3746" Oct 30 01:24:26.867016 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:26.866981 4517 scope.go:122] "RemoveContainer" containerID="57d08505b8895f9d4c1a2d73180ff44b9b58b895838b46f5ee82eebd4cad6b65" Oct 30 01:24:27.159602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:27.159491 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:28.842731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:28.842230 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:29.457542 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.457507 4517 generic.go:356] "Generic (PLEG): container finished" podID="35cf5570-1952-4038-af67-dded46578a90" containerID="2663e73863508e507556d1ccadf38601f3a46c545ca15add3e9a4b3220d715f1" exitCode=0 Oct 30 01:24:29.457780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.457588 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" event={"ID":"35cf5570-1952-4038-af67-dded46578a90","Type":"ContainerDied","Data":"2663e73863508e507556d1ccadf38601f3a46c545ca15add3e9a4b3220d715f1"} Oct 30 01:24:29.457780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.457673 4517 scope.go:122] "RemoveContainer" containerID="eb2158fbbbb8d4b5224b7904584a5a9e03eb1c35239ed3e6f4aad2aa1111ff4a" Oct 30 01:24:29.530070 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.530033 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:24:29.593887 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.593846 4517 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:24:29.594066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.593917 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:24:29.608620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.608571 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35cf5570-1952-4038-af67-dded46578a90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cf5570-1952-4038-af67-dded46578a90-istio-podinfo\") pod \"35cf5570-1952-4038-af67-dded46578a90\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " Oct 30 01:24:29.608838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.608648 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-socket\") pod \"35cf5570-1952-4038-af67-dded46578a90\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " Oct 30 01:24:29.608838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.608786 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35cf5570-1952-4038-af67-dded46578a90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cf5570-1952-4038-af67-dded46578a90-istiod-ca-cert\") pod \"35cf5570-1952-4038-af67-dded46578a90\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " Oct 30 01:24:29.608963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.608901 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-data\") pod \"35cf5570-1952-4038-af67-dded46578a90\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " Oct 30 01:24:29.608963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.608939 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-istio-token\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-istio-token\") pod \"35cf5570-1952-4038-af67-dded46578a90\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " Oct 30 01:24:29.609074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609015 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-credential-socket\") pod \"35cf5570-1952-4038-af67-dded46578a90\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " Oct 30 01:24:29.609074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609065 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-kube-api-access-z9xp7\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-kube-api-access-z9xp7\") pod \"35cf5570-1952-4038-af67-dded46578a90\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " Oct 30 01:24:29.609188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609099 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-envoy\") pod \"35cf5570-1952-4038-af67-dded46578a90\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " Oct 30 01:24:29.609188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609129 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-certs\") pod \"35cf5570-1952-4038-af67-dded46578a90\" (UID: \"35cf5570-1952-4038-af67-dded46578a90\") " Oct 30 01:24:29.609188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609146 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-socket" pod "35cf5570-1952-4038-af67-dded46578a90" (UID: "35cf5570-1952-4038-af67-dded46578a90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.609360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609314 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-credential-socket" pod "35cf5570-1952-4038-af67-dded46578a90" (UID: "35cf5570-1952-4038-af67-dded46578a90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.609360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609339 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cf5570-1952-4038-af67-dded46578a90-istiod-ca-cert" pod "35cf5570-1952-4038-af67-dded46578a90" (UID: "35cf5570-1952-4038-af67-dded46578a90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:29.609817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609541 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-certs" pod "35cf5570-1952-4038-af67-dded46578a90" (UID: "35cf5570-1952-4038-af67-dded46578a90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.609817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609558 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-data" pod "35cf5570-1952-4038-af67-dded46578a90" (UID: "35cf5570-1952-4038-af67-dded46578a90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.609817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609690 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cf5570-1952-4038-af67-dded46578a90-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.609817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609735 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.609817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609749 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.609817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609764 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.609817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.609793 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.611537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.611370 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-envoy" pod "35cf5570-1952-4038-af67-dded46578a90" (UID: "35cf5570-1952-4038-af67-dded46578a90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.611973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.611950 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35cf5570-1952-4038-af67-dded46578a90-istio-podinfo" pod "35cf5570-1952-4038-af67-dded46578a90" (UID: "35cf5570-1952-4038-af67-dded46578a90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:29.612098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.612054 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-istio-token" pod "35cf5570-1952-4038-af67-dded46578a90" (UID: "35cf5570-1952-4038-af67-dded46578a90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:29.612168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.612112 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-kube-api-access-z9xp7" pod "35cf5570-1952-4038-af67-dded46578a90" (UID: "35cf5570-1952-4038-af67-dded46578a90"). InnerVolumeSpecName "kube-api-access-z9xp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:29.710434 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.710397 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.710434 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.710429 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9xp7\" (UniqueName: \"kubernetes.io/projected/35cf5570-1952-4038-af67-dded46578a90-kube-api-access-z9xp7\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.710434 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.710440 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cf5570-1952-4038-af67-dded46578a90-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.710434 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:29.710448 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cf5570-1952-4038-af67-dded46578a90-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:30.263927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.263884 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn"] Oct 30 01:24:30.264580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264557 4517 cpu_manager.go:414] "containerMap: removing container" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" Oct 30 01:24:30.264580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264579 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264598 4517 cpu_manager.go:414] "containerMap: removing container" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264606 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264616 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f56f3a12-1267-487b-b597-ec9514c6a26d" containerName="revoke-keys" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264624 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56f3a12-1267-487b-b597-ec9514c6a26d" containerName="revoke-keys" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264637 4517 cpu_manager.go:414] "containerMap: removing container" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264644 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264657 4517 cpu_manager.go:414] "containerMap: removing container" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264665 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264698 4517 cpu_manager.go:414] "containerMap: removing container" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264723 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264738 4517 cpu_manager.go:414] "containerMap: removing container" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264745 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5e313a-1534-49cc-bb1c-9ef6a29649e1" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264854 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264866 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264877 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264888 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e1387e-0825-4d7d-b5c5-e1282d033a11" containerName="istio-proxy" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264897 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56f3a12-1267-487b-b597-ec9514c6a26d" containerName="revoke-keys" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264906 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3f002e-af8a-40f2-a644-59662eaff603" containerName="main" Oct 30 01:24:30.264937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.264914 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="340bbdbe-47e6-43d3-b384-e2f5a8e4b726" containerName="main" Oct 30 01:24:30.268862 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.268836 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.289679 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.289642 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn"] Oct 30 01:24:30.317696 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.317657 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed237c1-h87vn\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.317937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.317729 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed237c1-h87vn\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.317937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.317854 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr2cc\" (UniqueName: \"kubernetes.io/projected/3c5d9ba8-59b5-4e95-869f-c068941cafc0-kube-api-access-tr2cc\") pod \"maas-api-revoke-sub-7ed237c1-h87vn\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.363216 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.363172 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:24:30.363415 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.363325 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:24:30.418728 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.418679 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr2cc\" (UniqueName: \"kubernetes.io/projected/3c5d9ba8-59b5-4e95-869f-c068941cafc0-kube-api-access-tr2cc\") pod \"maas-api-revoke-sub-7ed237c1-h87vn\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.418938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.418832 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed237c1-h87vn\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.418938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.418859 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed237c1-h87vn\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.419819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.419784 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed237c1-h87vn\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.421512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.421489 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed237c1-h87vn\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.440944 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.440914 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr2cc\" (UniqueName: \"kubernetes.io/projected/3c5d9ba8-59b5-4e95-869f-c068941cafc0-kube-api-access-tr2cc\") pod \"maas-api-revoke-sub-7ed237c1-h87vn\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.465274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.465242 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" Oct 30 01:24:30.465274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.465243 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m" event={"ID":"35cf5570-1952-4038-af67-dded46578a90","Type":"ContainerDied","Data":"dd09f9eda20b040e5ab83d0990e761704505f42540f4792e49c883fbdac13584"} Oct 30 01:24:30.465547 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.465316 4517 scope.go:122] "RemoveContainer" containerID="2663e73863508e507556d1ccadf38601f3a46c545ca15add3e9a4b3220d715f1" Oct 30 01:24:30.513770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.513655 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m"] Oct 30 01:24:30.523952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.523915 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-qv88m"] Oct 30 01:24:30.590675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.590638 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:30.646339 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.646268 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:30.765141 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:24:30.765039 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:24:30.925768 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:30.925697 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn"] Oct 30 01:24:30.928969 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:24:30.928940 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5d9ba8_59b5_4e95_869f_c068941cafc0.slice/crio-a7b798112897aa3212642fcd6ec5fb174295e6bd2559e2abd74b48c8eec5377b WatchSource:0}: Error finding container a7b798112897aa3212642fcd6ec5fb174295e6bd2559e2abd74b48c8eec5377b: Status 404 returned error can't find the container with id a7b798112897aa3212642fcd6ec5fb174295e6bd2559e2abd74b48c8eec5377b Oct 30 01:24:31.475972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.475866 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" event={"ID":"3c5d9ba8-59b5-4e95-869f-c068941cafc0","Type":"ContainerStarted","Data":"b00e298927546acc6ed3a38bc9457eb9a35fae5a91859a58191983b21832078b"} Oct 30 01:24:31.475972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.475931 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" event={"ID":"3c5d9ba8-59b5-4e95-869f-c068941cafc0","Type":"ContainerStarted","Data":"a7b798112897aa3212642fcd6ec5fb174295e6bd2559e2abd74b48c8eec5377b"} Oct 30 01:24:31.513201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.513140 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" podStartSLOduration=1.51312658 podStartE2EDuration="1.51312658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:31.511973577 +0000 UTC m=+3009.424100744" watchObservedRunningTime="2026-10-30 01:24:31.51312658 +0000 UTC m=+3009.425253723" Oct 30 01:24:31.521859 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.521827 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:31.871114 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.871080 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 01:24:31.937130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937091 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-kube-api-access-fmwgk\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-kube-api-access-fmwgk\") pod \"e68d521f-b88b-40e6-8f15-f822d8394f55\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " Oct 30 01:24:31.937354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937160 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68d521f-b88b-40e6-8f15-f822d8394f55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68d521f-b88b-40e6-8f15-f822d8394f55-istiod-ca-cert\") pod \"e68d521f-b88b-40e6-8f15-f822d8394f55\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " Oct 30 01:24:31.937354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937208 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-credential-socket\") pod \"e68d521f-b88b-40e6-8f15-f822d8394f55\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " Oct 30 01:24:31.937354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937245 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-token\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-token\") pod \"e68d521f-b88b-40e6-8f15-f822d8394f55\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " Oct 30 01:24:31.937354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937296 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-envoy\") pod \"e68d521f-b88b-40e6-8f15-f822d8394f55\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " Oct 30 01:24:31.937354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937339 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-podinfo\") pod \"e68d521f-b88b-40e6-8f15-f822d8394f55\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " Oct 30 01:24:31.937668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937416 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-socket\") pod \"e68d521f-b88b-40e6-8f15-f822d8394f55\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " Oct 30 01:24:31.937668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937485 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-data\") pod \"e68d521f-b88b-40e6-8f15-f822d8394f55\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " Oct 30 01:24:31.937668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937522 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-certs\") pod \"e68d521f-b88b-40e6-8f15-f822d8394f55\" (UID: \"e68d521f-b88b-40e6-8f15-f822d8394f55\") " Oct 30 01:24:31.937668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937524 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-credential-socket" pod "e68d521f-b88b-40e6-8f15-f822d8394f55" (UID: "e68d521f-b88b-40e6-8f15-f822d8394f55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.937668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937617 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68d521f-b88b-40e6-8f15-f822d8394f55-istiod-ca-cert" pod "e68d521f-b88b-40e6-8f15-f822d8394f55" (UID: "e68d521f-b88b-40e6-8f15-f822d8394f55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:31.937983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937886 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-socket" pod "e68d521f-b88b-40e6-8f15-f822d8394f55" (UID: "e68d521f-b88b-40e6-8f15-f822d8394f55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.937983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.937897 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-data" pod "e68d521f-b88b-40e6-8f15-f822d8394f55" (UID: "e68d521f-b88b-40e6-8f15-f822d8394f55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.938143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.938111 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-certs" pod "e68d521f-b88b-40e6-8f15-f822d8394f55" (UID: "e68d521f-b88b-40e6-8f15-f822d8394f55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.938245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.938210 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68d521f-b88b-40e6-8f15-f822d8394f55-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.938245 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.938234 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.938370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.938249 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.938370 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.938264 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.939938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.939908 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-kube-api-access-fmwgk" pod "e68d521f-b88b-40e6-8f15-f822d8394f55" (UID: "e68d521f-b88b-40e6-8f15-f822d8394f55"). InnerVolumeSpecName "kube-api-access-fmwgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:31.940554 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.940520 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-podinfo" pod "e68d521f-b88b-40e6-8f15-f822d8394f55" (UID: "e68d521f-b88b-40e6-8f15-f822d8394f55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:31.940663 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.940588 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-envoy" pod "e68d521f-b88b-40e6-8f15-f822d8394f55" (UID: "e68d521f-b88b-40e6-8f15-f822d8394f55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.940663 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:31.940643 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-token" pod "e68d521f-b88b-40e6-8f15-f822d8394f55" (UID: "e68d521f-b88b-40e6-8f15-f822d8394f55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:32.039111 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.039075 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmwgk\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-kube-api-access-fmwgk\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.039111 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.039107 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.039111 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.039117 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.039375 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.039125 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68d521f-b88b-40e6-8f15-f822d8394f55-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.039375 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.039137 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68d521f-b88b-40e6-8f15-f822d8394f55-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.226330 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.226298 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:32.235529 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.235494 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35cf5570-1952-4038-af67-dded46578a90" path="/var/lib/kubelet/pods/35cf5570-1952-4038-af67-dded46578a90/volumes" Oct 30 01:24:32.486843 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.486699 4517 generic.go:356] "Generic (PLEG): container finished" podID="3c5d9ba8-59b5-4e95-869f-c068941cafc0" containerID="b00e298927546acc6ed3a38bc9457eb9a35fae5a91859a58191983b21832078b" exitCode=0 Oct 30 01:24:32.487335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.487273 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" event={"ID":"3c5d9ba8-59b5-4e95-869f-c068941cafc0","Type":"ContainerDied","Data":"b00e298927546acc6ed3a38bc9457eb9a35fae5a91859a58191983b21832078b"} Oct 30 01:24:32.489180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.489151 4517 generic.go:356] "Generic (PLEG): container finished" podID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerID="eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5" exitCode=0 Oct 30 01:24:32.489351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.489265 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" event={"ID":"e68d521f-b88b-40e6-8f15-f822d8394f55","Type":"ContainerDied","Data":"eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5"} Oct 30 01:24:32.489351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.489312 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" event={"ID":"e68d521f-b88b-40e6-8f15-f822d8394f55","Type":"ContainerDied","Data":"df85ff1f67fda9c3a4f10965c6e5f545cda17234015df531f7e58ec774d1bdd6"} Oct 30 01:24:32.489351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.489339 4517 scope.go:122] "RemoveContainer" containerID="eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5" Oct 30 01:24:32.489535 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.489462 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4" Oct 30 01:24:32.506217 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.506188 4517 scope.go:122] "RemoveContainer" containerID="eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5" Oct 30 01:24:32.506600 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:24:32.506574 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5\": container with ID starting with eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5 not found: ID does not exist" containerID="eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5" Oct 30 01:24:32.506727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.506610 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5"} err="failed to get container status \"eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5\": rpc error: code = NotFound desc = could not find container \"eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5\": container with ID starting with eabe043adcadef862fbd1ebfb141ee7ef6b4e3da7e32d74e7192cb8c45265de5 not found: ID does not exist" Oct 30 01:24:32.587036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.574793 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4"] Oct 30 01:24:32.587036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.585755 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-765q4"] Oct 30 01:24:32.642270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:32.642227 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:33.475096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.475056 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:33.656396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.656357 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:33.757430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.757345 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-openshift-ca\") pod \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " Oct 30 01:24:33.757430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.757383 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5d9ba8-59b5-4e95-869f-c068941cafc0-kube-api-access-tr2cc\" (UniqueName: \"kubernetes.io/projected/3c5d9ba8-59b5-4e95-869f-c068941cafc0-kube-api-access-tr2cc\") pod \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " Oct 30 01:24:33.757658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.757454 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-xks-ca\") pod \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\" (UID: \"3c5d9ba8-59b5-4e95-869f-c068941cafc0\") " Oct 30 01:24:33.757886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.757855 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-openshift-ca" pod "3c5d9ba8-59b5-4e95-869f-c068941cafc0" (UID: "3c5d9ba8-59b5-4e95-869f-c068941cafc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:33.759931 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.759895 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-xks-ca" pod "3c5d9ba8-59b5-4e95-869f-c068941cafc0" (UID: "3c5d9ba8-59b5-4e95-869f-c068941cafc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:33.760059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.759970 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5d9ba8-59b5-4e95-869f-c068941cafc0-kube-api-access-tr2cc" pod "3c5d9ba8-59b5-4e95-869f-c068941cafc0" (UID: "3c5d9ba8-59b5-4e95-869f-c068941cafc0"). InnerVolumeSpecName "kube-api-access-tr2cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:33.859164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.859113 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:33.859164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.859151 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr2cc\" (UniqueName: \"kubernetes.io/projected/3c5d9ba8-59b5-4e95-869f-c068941cafc0-kube-api-access-tr2cc\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:33.859164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.859162 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5d9ba8-59b5-4e95-869f-c068941cafc0-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:24:33.975193 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:33.975150 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:34.236097 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:34.236065 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" path="/var/lib/kubelet/pods/e68d521f-b88b-40e6-8f15-f822d8394f55/volumes" Oct 30 01:24:34.507169 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:34.507086 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" Oct 30 01:24:34.507169 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:34.507124 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn" event={"ID":"3c5d9ba8-59b5-4e95-869f-c068941cafc0","Type":"ContainerDied","Data":"a7b798112897aa3212642fcd6ec5fb174295e6bd2559e2abd74b48c8eec5377b"} Oct 30 01:24:34.507371 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:34.507176 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7b798112897aa3212642fcd6ec5fb174295e6bd2559e2abd74b48c8eec5377b" Oct 30 01:24:35.770779 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:35.770742 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:37.027746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:37.027693 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95"] Oct 30 01:24:37.037826 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:37.037791 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ebae79-grm95"] Oct 30 01:24:37.796670 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:37.796639 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:38.234828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:38.234743 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="923b47d8-5e17-4531-8114-12ccbc48e150" path="/var/lib/kubelet/pods/923b47d8-5e17-4531-8114-12ccbc48e150/volumes" Oct 30 01:24:39.356065 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:39.356023 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:39.559264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:39.559225 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:39.670224 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:39.670141 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:39.852804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:39.852769 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:39.971469 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:39.971371 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:40.082746 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:40.082676 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:40.262396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:40.262302 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:40.385627 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:40.385590 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:40.499024 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:40.498969 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:40.617999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:40.617955 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:41.893511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:41.893475 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:42.878359 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:42.878327 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:42.993879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:42.993848 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:43.629850 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:43.629804 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:43.766317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:43.766278 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:45.278141 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:45.278100 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:46.960654 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:46.960607 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:47.036938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:47.036893 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh"] Oct 30 01:24:47.047595 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:47.047548 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ddf68f-rgfwh"] Oct 30 01:24:47.954505 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:47.954467 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:48.090346 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:48.090307 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:48.236065 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:48.235971 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fff1ac-4baa-44b2-8e01-f4d2eded83bc" path="/var/lib/kubelet/pods/a5fff1ac-4baa-44b2-8e01-f4d2eded83bc/volumes" Oct 30 01:24:48.322156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:48.322116 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:48.732242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:48.732207 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:49.745985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:49.745942 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:49.963973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:49.963933 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:50.323909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:50.323873 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:50.935334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:50.935304 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:51.061976 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:51.061937 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:51.214909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:51.214809 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:51.609031 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:51.608992 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:53.479820 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:53.479784 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:55.219176 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:55.219138 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:56.918558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:56.918528 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:58.530773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:58.530737 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:58.721723 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:58.721667 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:58.872626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:58.872589 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz"] Oct 30 01:24:58.873207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:58.873171 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" containerID="cri-o://234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94" gracePeriod=30 Oct 30 01:24:58.936598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:58.936567 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:59.347308 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:59.347269 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:59.458147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:24:59.458110 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:00.531857 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:00.531822 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:00.764997 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:00.764944 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:25:02.179734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:02.179669 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:03.545563 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:03.545516 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:03.888909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:03.888827 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:04.026756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.023348 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:04.162653 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.162618 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:04.169193 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.169164 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 01:25:04.254859 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.254821 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39dadf0c-1fee-4fec-9080-c5816e329cde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dadf0c-1fee-4fec-9080-c5816e329cde-istiod-ca-cert\") pod \"39dadf0c-1fee-4fec-9080-c5816e329cde\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " Oct 30 01:25:04.255095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.254882 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-envoy\") pod \"39dadf0c-1fee-4fec-9080-c5816e329cde\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " Oct 30 01:25:04.255095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.254921 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-credential-socket\") pod \"39dadf0c-1fee-4fec-9080-c5816e329cde\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " Oct 30 01:25:04.255095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.255024 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-certs\") pod \"39dadf0c-1fee-4fec-9080-c5816e329cde\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " Oct 30 01:25:04.255095 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.255087 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-token\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-token\") pod \"39dadf0c-1fee-4fec-9080-c5816e329cde\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " Oct 30 01:25:04.255336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.255180 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-socket\") pod \"39dadf0c-1fee-4fec-9080-c5816e329cde\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " Oct 30 01:25:04.255336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.255220 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-podinfo\") pod \"39dadf0c-1fee-4fec-9080-c5816e329cde\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " Oct 30 01:25:04.255336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.255272 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-kube-api-access-nvr2k\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-kube-api-access-nvr2k\") pod \"39dadf0c-1fee-4fec-9080-c5816e329cde\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " Oct 30 01:25:04.255336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.255293 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39dadf0c-1fee-4fec-9080-c5816e329cde-istiod-ca-cert" pod "39dadf0c-1fee-4fec-9080-c5816e329cde" (UID: "39dadf0c-1fee-4fec-9080-c5816e329cde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:04.255336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.255322 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-data\") pod \"39dadf0c-1fee-4fec-9080-c5816e329cde\" (UID: \"39dadf0c-1fee-4fec-9080-c5816e329cde\") " Oct 30 01:25:04.259778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.255831 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dadf0c-1fee-4fec-9080-c5816e329cde-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.259778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.256277 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-data" pod "39dadf0c-1fee-4fec-9080-c5816e329cde" (UID: "39dadf0c-1fee-4fec-9080-c5816e329cde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.259778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.256483 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-certs" pod "39dadf0c-1fee-4fec-9080-c5816e329cde" (UID: "39dadf0c-1fee-4fec-9080-c5816e329cde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.259778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.256501 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-socket" pod "39dadf0c-1fee-4fec-9080-c5816e329cde" (UID: "39dadf0c-1fee-4fec-9080-c5816e329cde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.259778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.256727 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-credential-socket" pod "39dadf0c-1fee-4fec-9080-c5816e329cde" (UID: "39dadf0c-1fee-4fec-9080-c5816e329cde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.259778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.258111 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-envoy" pod "39dadf0c-1fee-4fec-9080-c5816e329cde" (UID: "39dadf0c-1fee-4fec-9080-c5816e329cde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.259778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.258866 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-token" pod "39dadf0c-1fee-4fec-9080-c5816e329cde" (UID: "39dadf0c-1fee-4fec-9080-c5816e329cde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:04.259778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.259267 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-podinfo" pod "39dadf0c-1fee-4fec-9080-c5816e329cde" (UID: "39dadf0c-1fee-4fec-9080-c5816e329cde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:04.259778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.259297 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-kube-api-access-nvr2k" pod "39dadf0c-1fee-4fec-9080-c5816e329cde" (UID: "39dadf0c-1fee-4fec-9080-c5816e329cde"). InnerVolumeSpecName "kube-api-access-nvr2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:04.356659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.356617 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.356659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.356654 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.356659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.356663 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.356963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.356671 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.356963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.356682 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.356963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.356689 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dadf0c-1fee-4fec-9080-c5816e329cde-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.356963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.356696 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dadf0c-1fee-4fec-9080-c5816e329cde-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.356963 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.356735 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvr2k\" (UniqueName: \"kubernetes.io/projected/39dadf0c-1fee-4fec-9080-c5816e329cde-kube-api-access-nvr2k\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.642287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.642251 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:04.735021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.734976 4517 generic.go:356] "Generic (PLEG): container finished" podID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerID="234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94" exitCode=0 Oct 30 01:25:04.735258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.735107 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" Oct 30 01:25:04.735258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.735117 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" event={"ID":"39dadf0c-1fee-4fec-9080-c5816e329cde","Type":"ContainerDied","Data":"234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94"} Oct 30 01:25:04.735258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.735177 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz" event={"ID":"39dadf0c-1fee-4fec-9080-c5816e329cde","Type":"ContainerDied","Data":"6b0e4ac015fe44b240c448fa9249a8d54f2e28803434a6239a009afe63a2de88"} Oct 30 01:25:04.735258 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.735200 4517 scope.go:122] "RemoveContainer" containerID="234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94" Oct 30 01:25:04.747546 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.747517 4517 scope.go:122] "RemoveContainer" containerID="234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94" Oct 30 01:25:04.747917 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:04.747892 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94\": container with ID starting with 234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94 not found: ID does not exist" containerID="234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94" Oct 30 01:25:04.747971 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.747930 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94"} err="failed to get container status \"234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94\": rpc error: code = NotFound desc = could not find container \"234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94\": container with ID starting with 234eb10f1ab4c8c371ed526e74e44ad0a932865f057a6139b123d6301f9abf94 not found: ID does not exist" Oct 30 01:25:04.763999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.763963 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:04.793061 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.793020 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz"] Oct 30 01:25:04.803474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.803423 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-tj8nz"] Oct 30 01:25:04.882924 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:04.882886 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:05.005283 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:05.005179 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:05.396288 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:05.396217 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:06.234110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:06.234072 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" path="/var/lib/kubelet/pods/39dadf0c-1fee-4fec-9080-c5816e329cde/volumes" Oct 30 01:25:06.600130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:06.600097 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:08.245758 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:08.245680 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:09.939022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:09.938978 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:11.034243 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:11.034206 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:11.146248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:11.146209 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:11.321951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:11.321850 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb"] Oct 30 01:25:11.322459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:11.322423 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" containerID="cri-o://660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3" gracePeriod=30 Oct 30 01:25:11.369949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:11.369913 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:11.562253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:11.562216 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:12.129606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:12.129563 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:12.245421 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:12.245375 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:12.472945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:12.472863 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:12.970402 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:12.970352 4517 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:25:12.970609 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:12.970442 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:25:13.532682 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:13.532647 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:15.323426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:15.323368 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:16.623345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.623314 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 01:25:16.713539 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713501 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-socket\") pod \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " Oct 30 01:25:16.713737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713557 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-podinfo\") pod \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " Oct 30 01:25:16.713737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713580 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-data\") pod \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " Oct 30 01:25:16.713737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713612 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istiod-ca-cert\") pod \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " Oct 30 01:25:16.713918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713743 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-certs\") pod \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " Oct 30 01:25:16.713918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713795 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-envoy\") pod \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " Oct 30 01:25:16.713918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713831 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-credential-socket\") pod \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " Oct 30 01:25:16.713918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713860 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-kube-api-access-f2m2z\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-kube-api-access-f2m2z\") pod \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " Oct 30 01:25:16.713918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713904 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-token\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-token\") pod \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\" (UID: \"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018\") " Oct 30 01:25:16.713918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713876 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-socket" pod "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" (UID: "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:16.714340 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.713938 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-data" pod "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" (UID: "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:16.714340 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.714194 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istiod-ca-cert" pod "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" (UID: "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:16.714452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.714413 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.714452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.714434 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.714452 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.714450 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.714570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.714424 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-credential-socket" pod "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" (UID: "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:16.714570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.714474 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-certs" pod "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" (UID: "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:16.716435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.716413 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-envoy" pod "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" (UID: "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:16.716660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.716642 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-token" pod "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" (UID: "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:16.716798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.716683 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-podinfo" pod "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" (UID: "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:16.716798 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.716730 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-kube-api-access-f2m2z" pod "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" (UID: "cde95c4e-9e6e-4c58-a66b-5b8f12aa0018"). InnerVolumeSpecName "kube-api-access-f2m2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:16.811821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.811784 4517 generic.go:356] "Generic (PLEG): container finished" podID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerID="660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3" exitCode=0 Oct 30 01:25:16.812022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.811856 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" Oct 30 01:25:16.812022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.811880 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" event={"ID":"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018","Type":"ContainerDied","Data":"660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3"} Oct 30 01:25:16.812022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.811942 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb" event={"ID":"cde95c4e-9e6e-4c58-a66b-5b8f12aa0018","Type":"ContainerDied","Data":"c40ebf7449b8166c125902d42cf1561563104225ca9bc33d9be04b1cae951304"} Oct 30 01:25:16.812022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.811966 4517 scope.go:122] "RemoveContainer" containerID="660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3" Oct 30 01:25:16.815007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.814988 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.815155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.815010 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2m2z\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-kube-api-access-f2m2z\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.815155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.815020 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.815155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.815029 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.815155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.815048 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.815155 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.815060 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.822742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.822716 4517 scope.go:122] "RemoveContainer" containerID="50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf" Oct 30 01:25:16.831420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.831405 4517 scope.go:122] "RemoveContainer" containerID="660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3" Oct 30 01:25:16.831695 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:16.831678 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3\": container with ID starting with 660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3 not found: ID does not exist" containerID="660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3" Oct 30 01:25:16.831778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.831733 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3"} err="failed to get container status \"660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3\": rpc error: code = NotFound desc = could not find container \"660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3\": container with ID starting with 660e9d2c7a298c6b58a23f62e54036e5709d80ed4cbc2fc2ff7f82c395e59ac3 not found: ID does not exist" Oct 30 01:25:16.831778 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.831758 4517 scope.go:122] "RemoveContainer" containerID="50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf" Oct 30 01:25:16.832010 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:16.831994 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf\": container with ID starting with 50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf not found: ID does not exist" containerID="50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf" Oct 30 01:25:16.832049 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.832016 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf"} err="failed to get container status \"50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf\": rpc error: code = NotFound desc = could not find container \"50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf\": container with ID starting with 50a0ff2cd9bd3b5e64e7efef43e4f47e0852d01b176d6d0712398d60bc054adf not found: ID does not exist" Oct 30 01:25:16.858519 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.858439 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb"] Oct 30 01:25:16.864433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.864403 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-tb7qb"] Oct 30 01:25:16.952033 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:16.951989 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:18.233982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:18.233943 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" path="/var/lib/kubelet/pods/cde95c4e-9e6e-4c58-a66b-5b8f12aa0018/volumes" Oct 30 01:25:18.590053 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:18.589999 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:19.416516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:19.416460 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:20.200490 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:20.200456 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:21.748977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:21.748934 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:22.317455 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:22.317420 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth"] Oct 30 01:25:22.343785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:22.343750 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps"] Oct 30 01:25:22.459205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:22.459158 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:22.569079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:22.568965 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:23.424175 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:23.424131 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:23.540017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:23.539986 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:23.655534 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:23.655487 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:25.114834 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:25.114796 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:26.754337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:26.754295 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:26.947854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:26.947822 4517 scope.go:122] "RemoveContainer" containerID="6ceade41acbe4d21d519b6a5244b6dec75da925b79ad318047e45c7dd39ac55c" Oct 30 01:25:26.960019 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:26.959992 4517 scope.go:122] "RemoveContainer" containerID="276d3bda6ccc0eb6b4de478b9a5f6f4b3a9844c0b95227208880e4fb184a037e" Oct 30 01:25:27.318035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:27.317967 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" podUID="94d6be96-d442-4e00-b51c-6bf15b11a4b6" containerName="payload-processing" containerID="cri-o://205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171" gracePeriod=25 Oct 30 01:25:27.344413 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:27.344373 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" podUID="072afb9f-694b-40a8-8715-543942df57cf" containerName="payload-pre-processing" containerID="cri-o://2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0" gracePeriod=25 Oct 30 01:25:28.054160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.054119 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz"] Oct 30 01:25:28.054953 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.054896 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" podUID="04819ea4-d56b-455b-813a-dc677d867e3c" containerName="maas-api" containerID="cri-o://91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698" gracePeriod=30 Oct 30 01:25:28.146928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.146879 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:28.347090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.347053 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 01:25:28.448395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.448366 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04819ea4-d56b-455b-813a-dc677d867e3c-kube-api-access-hgpd9\" (UniqueName: \"kubernetes.io/projected/04819ea4-d56b-455b-813a-dc677d867e3c-kube-api-access-hgpd9\") pod \"04819ea4-d56b-455b-813a-dc677d867e3c\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " Oct 30 01:25:28.448395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.448407 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-metrics-certs\") pod \"04819ea4-d56b-455b-813a-dc677d867e3c\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " Oct 30 01:25:28.448648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.448429 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-maas-api-tls\") pod \"04819ea4-d56b-455b-813a-dc677d867e3c\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " Oct 30 01:25:28.448648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.448603 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04819ea4-d56b-455b-813a-dc677d867e3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04819ea4-d56b-455b-813a-dc677d867e3c-openshift-service-ca\") pod \"04819ea4-d56b-455b-813a-dc677d867e3c\" (UID: \"04819ea4-d56b-455b-813a-dc677d867e3c\") " Oct 30 01:25:28.449071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.449039 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04819ea4-d56b-455b-813a-dc677d867e3c-openshift-service-ca" pod "04819ea4-d56b-455b-813a-dc677d867e3c" (UID: "04819ea4-d56b-455b-813a-dc677d867e3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:28.450881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.450826 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04819ea4-d56b-455b-813a-dc677d867e3c-kube-api-access-hgpd9" pod "04819ea4-d56b-455b-813a-dc677d867e3c" (UID: "04819ea4-d56b-455b-813a-dc677d867e3c"). InnerVolumeSpecName "kube-api-access-hgpd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:28.450881 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.450862 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-metrics-certs" pod "04819ea4-d56b-455b-813a-dc677d867e3c" (UID: "04819ea4-d56b-455b-813a-dc677d867e3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:28.451083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.450927 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-maas-api-tls" pod "04819ea4-d56b-455b-813a-dc677d867e3c" (UID: "04819ea4-d56b-455b-813a-dc677d867e3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:28.550462 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.550425 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04819ea4-d56b-455b-813a-dc677d867e3c-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:28.550462 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.550464 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgpd9\" (UniqueName: \"kubernetes.io/projected/04819ea4-d56b-455b-813a-dc677d867e3c-kube-api-access-hgpd9\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:28.550772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.550482 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:28.550772 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.550499 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04819ea4-d56b-455b-813a-dc677d867e3c-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:28.622412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.622294 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:28.750977 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.750940 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:28.897342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.897202 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:28.900401 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.900373 4517 generic.go:356] "Generic (PLEG): container finished" podID="04819ea4-d56b-455b-813a-dc677d867e3c" containerID="91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698" exitCode=0 Oct 30 01:25:28.900573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.900433 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" event={"ID":"04819ea4-d56b-455b-813a-dc677d867e3c","Type":"ContainerDied","Data":"91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698"} Oct 30 01:25:28.900573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.900489 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" Oct 30 01:25:28.900573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.900519 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz" event={"ID":"04819ea4-d56b-455b-813a-dc677d867e3c","Type":"ContainerDied","Data":"8b9ff7b2564d929f30377dd2890a8183812ea42d35eae0e0094238df26e38f99"} Oct 30 01:25:28.900573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.900547 4517 scope.go:122] "RemoveContainer" containerID="91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698" Oct 30 01:25:28.912138 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.912111 4517 scope.go:122] "RemoveContainer" containerID="91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698" Oct 30 01:25:28.912497 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:28.912472 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698\": container with ID starting with 91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698 not found: ID does not exist" containerID="91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698" Oct 30 01:25:28.912575 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.912508 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698"} err="failed to get container status \"91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698\": rpc error: code = NotFound desc = could not find container \"91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698\": container with ID starting with 91b515c74641ce66c2f0c18d7d62ba787d5b20947929490111f42023dd041698 not found: ID does not exist" Oct 30 01:25:28.950985 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.950941 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz"] Oct 30 01:25:28.959883 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:28.959847 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-16c1bb-7cb95c796b-tmkcz"] Oct 30 01:25:29.297658 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:29.297618 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:29.412440 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:29.412408 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:29.523545 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:29.523507 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:29.919342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:29.919296 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:30.234600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:30.234516 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04819ea4-d56b-455b-813a-dc677d867e3c" path="/var/lib/kubelet/pods/04819ea4-d56b-455b-813a-dc677d867e3c/volumes" Oct 30 01:25:30.760096 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:30.760056 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:25:30.861602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:30.861560 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" podUID="94d6be96-d442-4e00-b51c-6bf15b11a4b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:30.869169 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:30.869126 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" podUID="072afb9f-694b-40a8-8715-543942df57cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:31.135212 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:31.135171 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:32.729197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:32.729159 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:34.414113 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:34.414068 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:35.805878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:35.805844 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:35.918587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:35.918537 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:36.042508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:36.042467 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:36.082432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:36.082338 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n"] Oct 30 01:25:36.083010 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:36.082968 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" containerID="cri-o://8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe" gracePeriod=30 Oct 30 01:25:36.085904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:36.085861 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h"] Oct 30 01:25:36.086434 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:36.086393 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" containerID="cri-o://e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012" gracePeriod=30 Oct 30 01:25:36.155983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:36.155935 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:36.624083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:36.624045 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:36.816303 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:36.816248 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:36.937738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:36.937621 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:37.146261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.146231 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:37.646227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646189 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f"] Oct 30 01:25:37.646607 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646595 4517 cpu_manager.go:414] "containerMap: removing container" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" Oct 30 01:25:37.646655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646609 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" Oct 30 01:25:37.646655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646618 4517 cpu_manager.go:414] "containerMap: removing container" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" Oct 30 01:25:37.646655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646624 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cf5570-1952-4038-af67-dded46578a90" containerName="istio-proxy" Oct 30 01:25:37.646655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646636 4517 cpu_manager.go:414] "containerMap: removing container" podUID="04819ea4-d56b-455b-813a-dc677d867e3c" containerName="maas-api" Oct 30 01:25:37.646655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646641 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04819ea4-d56b-455b-813a-dc677d867e3c" containerName="maas-api" Oct 30 01:25:37.646655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646649 4517 cpu_manager.go:414] "containerMap: removing container" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" Oct 30 01:25:37.646655 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646653 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646662 4517 cpu_manager.go:414] "containerMap: removing container" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646666 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646679 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646684 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646702 4517 cpu_manager.go:414] "containerMap: removing container" podUID="3c5d9ba8-59b5-4e95-869f-c068941cafc0" containerName="revoke-keys" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646720 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5d9ba8-59b5-4e95-869f-c068941cafc0" containerName="revoke-keys" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646777 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646787 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="04819ea4-d56b-455b-813a-dc677d867e3c" containerName="maas-api" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646794 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde95c4e-9e6e-4c58-a66b-5b8f12aa0018" containerName="istio-proxy" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646800 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="39dadf0c-1fee-4fec-9080-c5816e329cde" containerName="istio-proxy" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646806 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68d521f-b88b-40e6-8f15-f822d8394f55" containerName="istio-proxy" Oct 30 01:25:37.646922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.646814 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5d9ba8-59b5-4e95-869f-c068941cafc0" containerName="revoke-keys" Oct 30 01:25:37.651426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.651403 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.677132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.677093 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:25:37.677317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.677257 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:25:37.679441 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.679417 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f"] Oct 30 01:25:37.750960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.750917 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlft8\" (UniqueName: \"kubernetes.io/projected/b10e2a7e-6528-41bf-a775-9fc836a39ba3-kube-api-access-wlft8\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.751203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.751123 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.751203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.751161 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.851823 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.851778 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlft8\" (UniqueName: \"kubernetes.io/projected/b10e2a7e-6528-41bf-a775-9fc836a39ba3-kube-api-access-wlft8\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.852345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.851947 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.852345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.851986 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.852750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.852727 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.854720 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.854679 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.873508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.873474 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlft8\" (UniqueName: \"kubernetes.io/projected/b10e2a7e-6528-41bf-a775-9fc836a39ba3-kube-api-access-wlft8\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:37.945194 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.945105 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:37.964660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:37.964617 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:38.278196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:38.278106 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f"] Oct 30 01:25:38.282438 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:25:38.282404 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10e2a7e_6528_41bf_a775_9fc836a39ba3.slice/crio-1253b370f5759e95c07063e33519f86222d90be6ce5dda34a363d69890b3a4a1 WatchSource:0}: Error finding container 1253b370f5759e95c07063e33519f86222d90be6ce5dda34a363d69890b3a4a1: Status 404 returned error can't find the container with id 1253b370f5759e95c07063e33519f86222d90be6ce5dda34a363d69890b3a4a1 Oct 30 01:25:38.960479 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:38.960378 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" event={"ID":"b10e2a7e-6528-41bf-a775-9fc836a39ba3","Type":"ContainerStarted","Data":"a1d9b9f8445043a1af449dadb729050fff3d0cc4554669ff95053407d795afa0"} Oct 30 01:25:38.960479 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:38.960446 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" event={"ID":"b10e2a7e-6528-41bf-a775-9fc836a39ba3","Type":"ContainerStarted","Data":"1253b370f5759e95c07063e33519f86222d90be6ce5dda34a363d69890b3a4a1"} Oct 30 01:25:38.998635 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:38.998551 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" podStartSLOduration=1.9985341540000001 podStartE2EDuration="1.998534154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:38.99678701 +0000 UTC m=+3076.908914151" watchObservedRunningTime="2026-10-30 01:25:38.998534154 +0000 UTC m=+3076.910661295" Oct 30 01:25:39.734549 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:39.734517 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:39.969353 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:39.969314 4517 generic.go:356] "Generic (PLEG): container finished" podID="b10e2a7e-6528-41bf-a775-9fc836a39ba3" containerID="a1d9b9f8445043a1af449dadb729050fff3d0cc4554669ff95053407d795afa0" exitCode=0 Oct 30 01:25:39.969804 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:39.969408 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" event={"ID":"b10e2a7e-6528-41bf-a775-9fc836a39ba3","Type":"ContainerDied","Data":"a1d9b9f8445043a1af449dadb729050fff3d0cc4554669ff95053407d795afa0"} Oct 30 01:25:40.861734 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:40.861668 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" podUID="94d6be96-d442-4e00-b51c-6bf15b11a4b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:40.869152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:40.869117 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" podUID="072afb9f-694b-40a8-8715-543942df57cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:41.233581 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.233472 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:41.370309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.370004 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:25:41.382937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.382845 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-xks-ca\") pod \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " Oct 30 01:25:41.382937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.382897 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-openshift-ca\") pod \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " Oct 30 01:25:41.383207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.383023 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10e2a7e-6528-41bf-a775-9fc836a39ba3-kube-api-access-wlft8\" (UniqueName: \"kubernetes.io/projected/b10e2a7e-6528-41bf-a775-9fc836a39ba3-kube-api-access-wlft8\") pod \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\" (UID: \"b10e2a7e-6528-41bf-a775-9fc836a39ba3\") " Oct 30 01:25:41.383499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.383462 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-openshift-ca" pod "b10e2a7e-6528-41bf-a775-9fc836a39ba3" (UID: "b10e2a7e-6528-41bf-a775-9fc836a39ba3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:41.385334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.385298 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-xks-ca" pod "b10e2a7e-6528-41bf-a775-9fc836a39ba3" (UID: "b10e2a7e-6528-41bf-a775-9fc836a39ba3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:41.385956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.385924 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10e2a7e-6528-41bf-a775-9fc836a39ba3-kube-api-access-wlft8" pod "b10e2a7e-6528-41bf-a775-9fc836a39ba3" (UID: "b10e2a7e-6528-41bf-a775-9fc836a39ba3"). InnerVolumeSpecName "kube-api-access-wlft8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:41.397201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.397167 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:25:41.414087 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.414045 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:41.484388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484353 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-credential-socket\") pod \"251c373b-a73a-4aef-b678-b579391a6b96\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " Oct 30 01:25:41.484620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484457 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/251c373b-a73a-4aef-b678-b579391a6b96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251c373b-a73a-4aef-b678-b579391a6b96-istio-podinfo\") pod \"251c373b-a73a-4aef-b678-b579391a6b96\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " Oct 30 01:25:41.484620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484520 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-kube-api-access-v6rds\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-kube-api-access-v6rds\") pod \"251c373b-a73a-4aef-b678-b579391a6b96\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " Oct 30 01:25:41.484620 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484557 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-certs\") pod \"251c373b-a73a-4aef-b678-b579391a6b96\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " Oct 30 01:25:41.484811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484668 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-envoy\") pod \"251c373b-a73a-4aef-b678-b579391a6b96\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " Oct 30 01:25:41.484811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484737 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-istio-token\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-istio-token\") pod \"251c373b-a73a-4aef-b678-b579391a6b96\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " Oct 30 01:25:41.484811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484696 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-credential-socket" pod "251c373b-a73a-4aef-b678-b579391a6b96" (UID: "251c373b-a73a-4aef-b678-b579391a6b96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.484811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484789 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-socket\") pod \"251c373b-a73a-4aef-b678-b579391a6b96\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " Oct 30 01:25:41.484949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484852 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251c373b-a73a-4aef-b678-b579391a6b96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251c373b-a73a-4aef-b678-b579391a6b96-istiod-ca-cert\") pod \"251c373b-a73a-4aef-b678-b579391a6b96\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " Oct 30 01:25:41.484949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.484887 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-data\") pod \"251c373b-a73a-4aef-b678-b579391a6b96\" (UID: \"251c373b-a73a-4aef-b678-b579391a6b96\") " Oct 30 01:25:41.485174 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.485159 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.485228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.485176 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.485228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.485185 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10e2a7e-6528-41bf-a775-9fc836a39ba3-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.485228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.485193 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlft8\" (UniqueName: \"kubernetes.io/projected/b10e2a7e-6528-41bf-a775-9fc836a39ba3-kube-api-access-wlft8\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.485930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.485481 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-certs" pod "251c373b-a73a-4aef-b678-b579391a6b96" (UID: "251c373b-a73a-4aef-b678-b579391a6b96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.485930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.485760 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-data" pod "251c373b-a73a-4aef-b678-b579391a6b96" (UID: "251c373b-a73a-4aef-b678-b579391a6b96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.485930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.485851 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251c373b-a73a-4aef-b678-b579391a6b96-istiod-ca-cert" pod "251c373b-a73a-4aef-b678-b579391a6b96" (UID: "251c373b-a73a-4aef-b678-b579391a6b96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:41.485930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.485895 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-socket" pod "251c373b-a73a-4aef-b678-b579391a6b96" (UID: "251c373b-a73a-4aef-b678-b579391a6b96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.488001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.487970 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-istio-token" pod "251c373b-a73a-4aef-b678-b579391a6b96" (UID: "251c373b-a73a-4aef-b678-b579391a6b96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:41.488126 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.487975 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/251c373b-a73a-4aef-b678-b579391a6b96-istio-podinfo" pod "251c373b-a73a-4aef-b678-b579391a6b96" (UID: "251c373b-a73a-4aef-b678-b579391a6b96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:41.488396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.488370 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-envoy" pod "251c373b-a73a-4aef-b678-b579391a6b96" (UID: "251c373b-a73a-4aef-b678-b579391a6b96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.488910 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.488884 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-kube-api-access-v6rds" pod "251c373b-a73a-4aef-b678-b579391a6b96" (UID: "251c373b-a73a-4aef-b678-b579391a6b96"). InnerVolumeSpecName "kube-api-access-v6rds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:41.577632 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.577585 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn"] Oct 30 01:25:41.585624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.585588 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q"] Oct 30 01:25:41.585867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.585825 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3488a16-3617-4db6-ae61-8c93a76f46e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3488a16-3617-4db6-ae61-8c93a76f46e1-istiod-ca-cert\") pod \"c3488a16-3617-4db6-ae61-8c93a76f46e1\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " Oct 30 01:25:41.585969 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.585890 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-certs\") pod \"c3488a16-3617-4db6-ae61-8c93a76f46e1\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " Oct 30 01:25:41.586039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.585978 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-kube-api-access-pmrjx\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-kube-api-access-pmrjx\") pod \"c3488a16-3617-4db6-ae61-8c93a76f46e1\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " Oct 30 01:25:41.586039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586030 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-socket\") pod \"c3488a16-3617-4db6-ae61-8c93a76f46e1\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " Oct 30 01:25:41.586149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586066 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-data\") pod \"c3488a16-3617-4db6-ae61-8c93a76f46e1\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " Oct 30 01:25:41.586218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586159 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-podinfo\") pod \"c3488a16-3617-4db6-ae61-8c93a76f46e1\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " Oct 30 01:25:41.586218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586209 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-credential-socket\") pod \"c3488a16-3617-4db6-ae61-8c93a76f46e1\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " Oct 30 01:25:41.586400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586241 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-envoy\") pod \"c3488a16-3617-4db6-ae61-8c93a76f46e1\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " Oct 30 01:25:41.586400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586281 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-token\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-token\") pod \"c3488a16-3617-4db6-ae61-8c93a76f46e1\" (UID: \"c3488a16-3617-4db6-ae61-8c93a76f46e1\") " Oct 30 01:25:41.586400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586307 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3488a16-3617-4db6-ae61-8c93a76f46e1-istiod-ca-cert" pod "c3488a16-3617-4db6-ae61-8c93a76f46e1" (UID: "c3488a16-3617-4db6-ae61-8c93a76f46e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:41.586634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586606 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-data" pod "c3488a16-3617-4db6-ae61-8c93a76f46e1" (UID: "c3488a16-3617-4db6-ae61-8c93a76f46e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586680 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251c373b-a73a-4aef-b678-b579391a6b96-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586701 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586826 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-certs" pod "c3488a16-3617-4db6-ae61-8c93a76f46e1" (UID: "c3488a16-3617-4db6-ae61-8c93a76f46e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586837 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251c373b-a73a-4aef-b678-b579391a6b96-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586852 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6rds\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-kube-api-access-v6rds\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586869 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586885 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586898 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251c373b-a73a-4aef-b678-b579391a6b96-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586910 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3488a16-3617-4db6-ae61-8c93a76f46e1-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.586922 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251c373b-a73a-4aef-b678-b579391a6b96-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.587014 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-socket" pod "c3488a16-3617-4db6-ae61-8c93a76f46e1" (UID: "c3488a16-3617-4db6-ae61-8c93a76f46e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.587336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.587301 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-credential-socket" pod "c3488a16-3617-4db6-ae61-8c93a76f46e1" (UID: "c3488a16-3617-4db6-ae61-8c93a76f46e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.590060 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.590029 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-kube-api-access-pmrjx" pod "c3488a16-3617-4db6-ae61-8c93a76f46e1" (UID: "c3488a16-3617-4db6-ae61-8c93a76f46e1"). InnerVolumeSpecName "kube-api-access-pmrjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:41.590208 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.590187 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-envoy" pod "c3488a16-3617-4db6-ae61-8c93a76f46e1" (UID: "c3488a16-3617-4db6-ae61-8c93a76f46e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:41.590292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.590233 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-token" pod "c3488a16-3617-4db6-ae61-8c93a76f46e1" (UID: "c3488a16-3617-4db6-ae61-8c93a76f46e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:41.595423 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.595389 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c"] Oct 30 01:25:41.596898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.596865 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-podinfo" pod "c3488a16-3617-4db6-ae61-8c93a76f46e1" (UID: "c3488a16-3617-4db6-ae61-8c93a76f46e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:41.687950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.687855 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.687950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.687891 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrjx\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-kube-api-access-pmrjx\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.687950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.687903 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.687950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.687911 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.687950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.687920 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.687950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.687928 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.687950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.687935 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.687950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.687943 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3488a16-3617-4db6-ae61-8c93a76f46e1-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:41.704616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.704526 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:41.982879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.982772 4517 generic.go:356] "Generic (PLEG): container finished" podID="251c373b-a73a-4aef-b678-b579391a6b96" containerID="e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012" exitCode=0 Oct 30 01:25:41.982879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.982844 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" Oct 30 01:25:41.983160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.982887 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" event={"ID":"251c373b-a73a-4aef-b678-b579391a6b96","Type":"ContainerDied","Data":"e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012"} Oct 30 01:25:41.983160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.982947 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h" event={"ID":"251c373b-a73a-4aef-b678-b579391a6b96","Type":"ContainerDied","Data":"845717606f22ad6da0a0b62a063ffe209c79fecca67257e00fb956f849a31df9"} Oct 30 01:25:41.983160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.982989 4517 scope.go:122] "RemoveContainer" containerID="e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012" Oct 30 01:25:41.987753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.986216 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" Oct 30 01:25:41.987753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.986249 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f" event={"ID":"b10e2a7e-6528-41bf-a775-9fc836a39ba3","Type":"ContainerDied","Data":"1253b370f5759e95c07063e33519f86222d90be6ce5dda34a363d69890b3a4a1"} Oct 30 01:25:41.987753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.986312 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1253b370f5759e95c07063e33519f86222d90be6ce5dda34a363d69890b3a4a1" Oct 30 01:25:41.988513 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.988481 4517 generic.go:356] "Generic (PLEG): container finished" podID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerID="8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe" exitCode=0 Oct 30 01:25:41.988638 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.988570 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" event={"ID":"c3488a16-3617-4db6-ae61-8c93a76f46e1","Type":"ContainerDied","Data":"8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe"} Oct 30 01:25:41.988638 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.988620 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" event={"ID":"c3488a16-3617-4db6-ae61-8c93a76f46e1","Type":"ContainerDied","Data":"22e9743c8f7af7424d3bdce7f56b81e0599bd43855929135c3832eec3a184261"} Oct 30 01:25:41.988788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.988735 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n" Oct 30 01:25:41.999986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.998762 4517 scope.go:122] "RemoveContainer" containerID="e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012" Oct 30 01:25:41.999986 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:41.999767 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012\": container with ID starting with e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012 not found: ID does not exist" containerID="e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012" Oct 30 01:25:41.999986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.999811 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012"} err="failed to get container status \"e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012\": rpc error: code = NotFound desc = could not find container \"e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012\": container with ID starting with e151d76e8379efafe98814ed626eee8687a41c1332d5c01cd198491e31fa3012 not found: ID does not exist" Oct 30 01:25:41.999986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:41.999841 4517 scope.go:122] "RemoveContainer" containerID="8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe" Oct 30 01:25:42.017427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:42.017400 4517 scope.go:122] "RemoveContainer" containerID="8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe" Oct 30 01:25:42.017827 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:42.017805 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe\": container with ID starting with 8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe not found: ID does not exist" containerID="8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe" Oct 30 01:25:42.017927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:42.017838 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe"} err="failed to get container status \"8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe\": rpc error: code = NotFound desc = could not find container \"8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe\": container with ID starting with 8034662446b6c2269dbc43bcb0d5864fd349160a92d2ed7365c03a415c114ffe not found: ID does not exist" Oct 30 01:25:42.050393 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:42.050355 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h"] Oct 30 01:25:42.055782 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:42.055745 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-dll5h"] Oct 30 01:25:42.085857 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:42.085817 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n"] Oct 30 01:25:42.091530 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:42.091493 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2qx9n"] Oct 30 01:25:42.234261 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:42.234175 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251c373b-a73a-4aef-b678-b579391a6b96" path="/var/lib/kubelet/pods/251c373b-a73a-4aef-b678-b579391a6b96/volumes" Oct 30 01:25:42.234664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:42.234645 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" path="/var/lib/kubelet/pods/c3488a16-3617-4db6-ae61-8c93a76f46e1/volumes" Oct 30 01:25:43.050255 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:43.050218 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:44.742770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:44.742699 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:46.422164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:46.422126 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:47.323867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.323820 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:47.438310 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.438280 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:47.622770 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.622742 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 01:25:47.648404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.648379 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 01:25:47.660035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.660011 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d6be96-d442-4e00-b51c-6bf15b11a4b6-kube-api-access-jjlcd\" (UniqueName: \"kubernetes.io/projected/94d6be96-d442-4e00-b51c-6bf15b11a4b6-kube-api-access-jjlcd\") pod \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\" (UID: \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\") " Oct 30 01:25:47.660149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.660132 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d6be96-d442-4e00-b51c-6bf15b11a4b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d6be96-d442-4e00-b51c-6bf15b11a4b6-plugins-config-volume\") pod \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\" (UID: \"94d6be96-d442-4e00-b51c-6bf15b11a4b6\") " Oct 30 01:25:47.660561 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.660540 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d6be96-d442-4e00-b51c-6bf15b11a4b6-plugins-config-volume" pod "94d6be96-d442-4e00-b51c-6bf15b11a4b6" (UID: "94d6be96-d442-4e00-b51c-6bf15b11a4b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:47.662527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.662352 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d6be96-d442-4e00-b51c-6bf15b11a4b6-kube-api-access-jjlcd" pod "94d6be96-d442-4e00-b51c-6bf15b11a4b6" (UID: "94d6be96-d442-4e00-b51c-6bf15b11a4b6"). InnerVolumeSpecName "kube-api-access-jjlcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:47.761896 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.761861 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072afb9f-694b-40a8-8715-543942df57cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072afb9f-694b-40a8-8715-543942df57cf-plugins-config-volume\") pod \"072afb9f-694b-40a8-8715-543942df57cf\" (UID: \"072afb9f-694b-40a8-8715-543942df57cf\") " Oct 30 01:25:47.762296 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.762054 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072afb9f-694b-40a8-8715-543942df57cf-kube-api-access-c7fxr\" (UniqueName: \"kubernetes.io/projected/072afb9f-694b-40a8-8715-543942df57cf-kube-api-access-c7fxr\") pod \"072afb9f-694b-40a8-8715-543942df57cf\" (UID: \"072afb9f-694b-40a8-8715-543942df57cf\") " Oct 30 01:25:47.762384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.762336 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072afb9f-694b-40a8-8715-543942df57cf-plugins-config-volume" pod "072afb9f-694b-40a8-8715-543942df57cf" (UID: "072afb9f-694b-40a8-8715-543942df57cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:47.762447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.762428 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjlcd\" (UniqueName: \"kubernetes.io/projected/94d6be96-d442-4e00-b51c-6bf15b11a4b6-kube-api-access-jjlcd\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.762518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.762446 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072afb9f-694b-40a8-8715-543942df57cf-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.762518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.762461 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d6be96-d442-4e00-b51c-6bf15b11a4b6-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:47.764900 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.764861 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072afb9f-694b-40a8-8715-543942df57cf-kube-api-access-c7fxr" pod "072afb9f-694b-40a8-8715-543942df57cf" (UID: "072afb9f-694b-40a8-8715-543942df57cf"). InnerVolumeSpecName "kube-api-access-c7fxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:47.864135 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:47.864043 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7fxr\" (UniqueName: \"kubernetes.io/projected/072afb9f-694b-40a8-8715-543942df57cf-kube-api-access-c7fxr\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:48.030514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.030478 4517 generic.go:356] "Generic (PLEG): container finished" podID="94d6be96-d442-4e00-b51c-6bf15b11a4b6" containerID="205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171" exitCode=0 Oct 30 01:25:48.030766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.030529 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" Oct 30 01:25:48.030766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.030568 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" event={"ID":"94d6be96-d442-4e00-b51c-6bf15b11a4b6","Type":"ContainerDied","Data":"205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171"} Oct 30 01:25:48.030766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.030634 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth" event={"ID":"94d6be96-d442-4e00-b51c-6bf15b11a4b6","Type":"ContainerDied","Data":"b0f0252865071cf263363cffa8acb0452703af8cd5471de0511669707a364a9c"} Oct 30 01:25:48.030766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.030670 4517 scope.go:122] "RemoveContainer" containerID="205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171" Oct 30 01:25:48.032123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.032100 4517 generic.go:356] "Generic (PLEG): container finished" podID="072afb9f-694b-40a8-8715-543942df57cf" containerID="2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0" exitCode=0 Oct 30 01:25:48.032236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.032157 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" Oct 30 01:25:48.032236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.032191 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" event={"ID":"072afb9f-694b-40a8-8715-543942df57cf","Type":"ContainerDied","Data":"2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0"} Oct 30 01:25:48.032236 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.032232 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps" event={"ID":"072afb9f-694b-40a8-8715-543942df57cf","Type":"ContainerDied","Data":"ab52654e1a1d4e7983d362e75eff0f6655ea2e8c50e0b9dab252e18aac34e500"} Oct 30 01:25:48.038474 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.038443 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:48.042877 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.042845 4517 scope.go:122] "RemoveContainer" containerID="205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171" Oct 30 01:25:48.043243 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:48.043215 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171\": container with ID starting with 205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171 not found: ID does not exist" containerID="205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171" Oct 30 01:25:48.043378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.043256 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171"} err="failed to get container status \"205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171\": rpc error: code = NotFound desc = could not find container \"205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171\": container with ID starting with 205d615c4309ce158af6c1b18222a4420afdb419ac3297b85c890c258ec18171 not found: ID does not exist" Oct 30 01:25:48.043378 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.043280 4517 scope.go:122] "RemoveContainer" containerID="2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0" Oct 30 01:25:48.052577 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.052555 4517 scope.go:122] "RemoveContainer" containerID="2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0" Oct 30 01:25:48.052913 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:48.052895 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0\": container with ID starting with 2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0 not found: ID does not exist" containerID="2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0" Oct 30 01:25:48.052981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.052923 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0"} err="failed to get container status \"2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0\": rpc error: code = NotFound desc = could not find container \"2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0\": container with ID starting with 2204c9a8c52d833a5b99d58d2b262b806837a71f246d95170eab59feaf3882d0 not found: ID does not exist" Oct 30 01:25:48.078517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.078480 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth"] Oct 30 01:25:48.088602 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.088571 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-16c1bb-59d75f5c74-v4bth"] Oct 30 01:25:48.127061 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.126867 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps"] Oct 30 01:25:48.130986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.130945 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-16c1bb-74c9779647-bvlps"] Oct 30 01:25:48.156631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.156596 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:48.233656 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.233604 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072afb9f-694b-40a8-8715-543942df57cf" path="/var/lib/kubelet/pods/072afb9f-694b-40a8-8715-543942df57cf/volumes" Oct 30 01:25:48.234164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:48.234141 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d6be96-d442-4e00-b51c-6bf15b11a4b6" path="/var/lib/kubelet/pods/94d6be96-d442-4e00-b51c-6bf15b11a4b6/volumes" Oct 30 01:25:49.553151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:49.553112 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:50.024524 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:50.024443 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:50.139168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:50.139125 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f"] Oct 30 01:25:50.141900 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:50.141875 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:50.149094 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:50.149067 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-8e24a5-9hg9f"] Oct 30 01:25:50.236246 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:50.236211 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10e2a7e-6528-41bf-a775-9fc836a39ba3" path="/var/lib/kubelet/pods/b10e2a7e-6528-41bf-a775-9fc836a39ba3/volumes" Oct 30 01:25:50.642764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:50.642727 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:52.006215 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:52.006178 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:52.039507 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:52.039438 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927"] Oct 30 01:25:52.042930 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:52.042891 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceed4a7-q8927"] Oct 30 01:25:52.234938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:52.234900 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece50684-1c45-48f6-b3a7-23f2aa6e888d" path="/var/lib/kubelet/pods/ece50684-1c45-48f6-b3a7-23f2aa6e888d/volumes" Oct 30 01:25:53.027007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:53.026965 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9"] Oct 30 01:25:53.037251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:53.037208 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a5697-kcnb9"] Oct 30 01:25:53.747866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:53.747831 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:54.246639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:54.246586 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17b1bc9-1b6b-499b-857c-994bb6283d0a" path="/var/lib/kubelet/pods/d17b1bc9-1b6b-499b-857c-994bb6283d0a/volumes" Oct 30 01:25:55.397134 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:55.397057 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:56.604595 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.604549 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" containerID="cri-o://b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618" gracePeriod=45 Oct 30 01:25:56.612072 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.612018 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" containerID="cri-o://c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60" gracePeriod=45 Oct 30 01:25:56.618308 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.618268 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" containerID="cri-o://a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51" gracePeriod=45 Oct 30 01:25:56.901952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.901892 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 01:25:56.941975 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.941947 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 01:25:56.950955 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.950924 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 01:25:56.967896 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.967859 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:56.975233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.975168 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-dshm\") pod \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " Oct 30 01:25:56.975442 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.975244 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-model-cache\") pod \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " Oct 30 01:25:56.975697 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.975667 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-model-cache" pod "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" (UID: "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:56.976232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.976208 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tmp-dir\") pod \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " Oct 30 01:25:56.976342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.976266 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-kube-api-access-t4nfl\" (UniqueName: \"kubernetes.io/projected/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-kube-api-access-t4nfl\") pod \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " Oct 30 01:25:56.976342 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.976299 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tls-certs\") pod \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " Oct 30 01:25:56.976460 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.976339 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-home\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-home\") pod \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\" (UID: \"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e\") " Oct 30 01:25:56.976509 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.976489 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tmp-dir" pod "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" (UID: "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:56.976733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.976698 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-home" pod "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" (UID: "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:56.976842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.976822 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:56.976909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.976848 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:56.978153 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.978127 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-dshm" pod "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" (UID: "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:56.978432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.978414 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-kube-api-access-t4nfl" pod "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" (UID: "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e"). InnerVolumeSpecName "kube-api-access-t4nfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:56.978815 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:56.978793 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tls-certs" pod "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" (UID: "bacc172f-6097-4a9d-bf9e-cbc32a88bf4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:57.078228 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078176 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tls-certs\") pod \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " Oct 30 01:25:57.078454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078255 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-home\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-home\") pod \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " Oct 30 01:25:57.078454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078332 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-model-cache\") pod \"f1751c25-d9a7-4672-8ca5-e1fafb277608\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " Oct 30 01:25:57.078454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078382 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-tmp-dir\") pod \"f1751c25-d9a7-4672-8ca5-e1fafb277608\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " Oct 30 01:25:57.078656 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078423 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-dshm\") pod \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " Oct 30 01:25:57.078656 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078519 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1751c25-d9a7-4672-8ca5-e1fafb277608-kube-api-access-xrznk\" (UniqueName: \"kubernetes.io/projected/f1751c25-d9a7-4672-8ca5-e1fafb277608-kube-api-access-xrznk\") pod \"f1751c25-d9a7-4672-8ca5-e1fafb277608\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " Oct 30 01:25:57.078656 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078595 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-dshm\") pod \"f1751c25-d9a7-4672-8ca5-e1fafb277608\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " Oct 30 01:25:57.078853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078657 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fabd8b-f688-4146-bc0d-742d7c1953ff-kube-api-access-m5lpk\" (UniqueName: \"kubernetes.io/projected/d8fabd8b-f688-4146-bc0d-742d7c1953ff-kube-api-access-m5lpk\") pod \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " Oct 30 01:25:57.078853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078658 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-model-cache" pod "f1751c25-d9a7-4672-8ca5-e1fafb277608" (UID: "f1751c25-d9a7-4672-8ca5-e1fafb277608"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.078853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078732 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tmp-dir\") pod \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " Oct 30 01:25:57.079017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078885 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-model-cache\") pod \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\" (UID: \"d8fabd8b-f688-4146-bc0d-742d7c1953ff\") " Oct 30 01:25:57.079017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078931 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-home\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-home\") pod \"f1751c25-d9a7-4672-8ca5-e1fafb277608\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " Oct 30 01:25:57.079017 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.078996 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1751c25-d9a7-4672-8ca5-e1fafb277608-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1751c25-d9a7-4672-8ca5-e1fafb277608-tls-certs\") pod \"f1751c25-d9a7-4672-8ca5-e1fafb277608\" (UID: \"f1751c25-d9a7-4672-8ca5-e1fafb277608\") " Oct 30 01:25:57.080732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.079619 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4nfl\" (UniqueName: \"kubernetes.io/projected/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-kube-api-access-t4nfl\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.080732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.079657 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.080732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.079677 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.080732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.079693 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.080732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.079755 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.080732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.079781 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-model-cache" pod "d8fabd8b-f688-4146-bc0d-742d7c1953ff" (UID: "d8fabd8b-f688-4146-bc0d-742d7c1953ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.080732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.080160 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tmp-dir" pod "d8fabd8b-f688-4146-bc0d-742d7c1953ff" (UID: "d8fabd8b-f688-4146-bc0d-742d7c1953ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.080732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.080474 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-home" pod "d8fabd8b-f688-4146-bc0d-742d7c1953ff" (UID: "d8fabd8b-f688-4146-bc0d-742d7c1953ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.081358 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.080776 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-tmp-dir" pod "f1751c25-d9a7-4672-8ca5-e1fafb277608" (UID: "f1751c25-d9a7-4672-8ca5-e1fafb277608"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.081358 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.080954 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-home" pod "f1751c25-d9a7-4672-8ca5-e1fafb277608" (UID: "f1751c25-d9a7-4672-8ca5-e1fafb277608"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.084548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.084500 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tls-certs" pod "d8fabd8b-f688-4146-bc0d-742d7c1953ff" (UID: "d8fabd8b-f688-4146-bc0d-742d7c1953ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:57.085492 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.085446 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1751c25-d9a7-4672-8ca5-e1fafb277608-tls-certs" pod "f1751c25-d9a7-4672-8ca5-e1fafb277608" (UID: "f1751c25-d9a7-4672-8ca5-e1fafb277608"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:57.085753 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.085675 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:57.086982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.086120 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fabd8b-f688-4146-bc0d-742d7c1953ff-kube-api-access-m5lpk" pod "d8fabd8b-f688-4146-bc0d-742d7c1953ff" (UID: "d8fabd8b-f688-4146-bc0d-742d7c1953ff"). InnerVolumeSpecName "kube-api-access-m5lpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:57.086982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.086282 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-dshm" pod "d8fabd8b-f688-4146-bc0d-742d7c1953ff" (UID: "d8fabd8b-f688-4146-bc0d-742d7c1953ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.086982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.086357 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1751c25-d9a7-4672-8ca5-e1fafb277608-kube-api-access-xrznk" pod "f1751c25-d9a7-4672-8ca5-e1fafb277608" (UID: "f1751c25-d9a7-4672-8ca5-e1fafb277608"). InnerVolumeSpecName "kube-api-access-xrznk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:57.086982 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.086873 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-dshm" pod "f1751c25-d9a7-4672-8ca5-e1fafb277608" (UID: "f1751c25-d9a7-4672-8ca5-e1fafb277608"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.096950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.096872 4517 generic.go:356] "Generic (PLEG): container finished" podID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerID="a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51" exitCode=0 Oct 30 01:25:57.096950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.096936 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" Oct 30 01:25:57.097312 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.097050 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" event={"ID":"f1751c25-d9a7-4672-8ca5-e1fafb277608","Type":"ContainerDied","Data":"a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51"} Oct 30 01:25:57.097312 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.097091 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c" event={"ID":"f1751c25-d9a7-4672-8ca5-e1fafb277608","Type":"ContainerDied","Data":"f3acb22c0492c9795f16fd231384b0fad891d3070cc805bd3e9058496f7c6920"} Oct 30 01:25:57.097312 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.097116 4517 scope.go:122] "RemoveContainer" containerID="a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51" Oct 30 01:25:57.100925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.100891 4517 generic.go:356] "Generic (PLEG): container finished" podID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerID="b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618" exitCode=0 Oct 30 01:25:57.101105 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.101041 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" Oct 30 01:25:57.101188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.101100 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" event={"ID":"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e","Type":"ContainerDied","Data":"b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618"} Oct 30 01:25:57.101188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.101177 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn" event={"ID":"bacc172f-6097-4a9d-bf9e-cbc32a88bf4e","Type":"ContainerDied","Data":"2da28cc53a2adf0b7773362a03be7ff69f5e688df187656d54b380bc31b5b046"} Oct 30 01:25:57.107813 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.107757 4517 generic.go:356] "Generic (PLEG): container finished" podID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerID="c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60" exitCode=0 Oct 30 01:25:57.108000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.107900 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" event={"ID":"d8fabd8b-f688-4146-bc0d-742d7c1953ff","Type":"ContainerDied","Data":"c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60"} Oct 30 01:25:57.108000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.107937 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" event={"ID":"d8fabd8b-f688-4146-bc0d-742d7c1953ff","Type":"ContainerDied","Data":"bad8d07937ccdf054c3fc0887b85b4b0e6c20b09e081a66a4968ddc3fb23ba54"} Oct 30 01:25:57.108000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.107994 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" Oct 30 01:25:57.113927 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.113899 4517 scope.go:122] "RemoveContainer" containerID="a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51" Oct 30 01:25:57.114339 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:57.114311 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51\": container with ID starting with a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51 not found: ID does not exist" containerID="a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51" Oct 30 01:25:57.114410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.114355 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51"} err="failed to get container status \"a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51\": rpc error: code = NotFound desc = could not find container \"a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51\": container with ID starting with a7b4b4bc6e72b932d3773954693ec128ffb0f894282b357698e728f2edde3e51 not found: ID does not exist" Oct 30 01:25:57.114410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.114383 4517 scope.go:122] "RemoveContainer" containerID="b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618" Oct 30 01:25:57.127051 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.127025 4517 scope.go:122] "RemoveContainer" containerID="b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618" Oct 30 01:25:57.127488 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:57.127460 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618\": container with ID starting with b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618 not found: ID does not exist" containerID="b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618" Oct 30 01:25:57.127624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.127493 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618"} err="failed to get container status \"b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618\": rpc error: code = NotFound desc = could not find container \"b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618\": container with ID starting with b0c79da12c802cbd1a376e86d95d90ae8baff2f510bc8778c1891e3a63800618 not found: ID does not exist" Oct 30 01:25:57.127624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.127515 4517 scope.go:122] "RemoveContainer" containerID="c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60" Oct 30 01:25:57.137278 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.137253 4517 scope.go:122] "RemoveContainer" containerID="c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60" Oct 30 01:25:57.137618 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:25:57.137596 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60\": container with ID starting with c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60 not found: ID does not exist" containerID="c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60" Oct 30 01:25:57.137735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.137648 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60"} err="failed to get container status \"c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60\": rpc error: code = NotFound desc = could not find container \"c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60\": container with ID starting with c593de39fbb7d0fb3093bc50ceceeac5ef7053914c115c0c06bf00ef69533a60 not found: ID does not exist" Oct 30 01:25:57.174218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.174173 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c"] Oct 30 01:25:57.180025 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.179981 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/simulator-w1-beb8d5-kserve-6959849856-6wq7c"] Oct 30 01:25:57.180388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180368 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180393 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5lpk\" (UniqueName: \"kubernetes.io/projected/d8fabd8b-f688-4146-bc0d-742d7c1953ff-kube-api-access-m5lpk\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180406 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180418 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180427 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180434 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1751c25-d9a7-4672-8ca5-e1fafb277608-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180441 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fabd8b-f688-4146-bc0d-742d7c1953ff-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180449 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180454 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180456 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1751c25-d9a7-4672-8ca5-e1fafb277608-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180463 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fabd8b-f688-4146-bc0d-742d7c1953ff-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.180764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.180470 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrznk\" (UniqueName: \"kubernetes.io/projected/f1751c25-d9a7-4672-8ca5-e1fafb277608-kube-api-access-xrznk\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.210688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.210650 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q"] Oct 30 01:25:57.215090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.215056 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q"] Oct 30 01:25:57.252576 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.252528 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn"] Oct 30 01:25:57.259941 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.259903 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-w1-beb8d5-kserve-5476dc9844-mr4sn"] Oct 30 01:25:57.307527 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.307473 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:57.753090 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.753049 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:57.832725 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:57.832664 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/premium-w1-beb8d5-kserve-764dff75cc-5ts8q" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 01:25:58.234213 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:58.234173 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" path="/var/lib/kubelet/pods/bacc172f-6097-4a9d-bf9e-cbc32a88bf4e/volumes" Oct 30 01:25:58.234673 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:58.234635 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" path="/var/lib/kubelet/pods/d8fabd8b-f688-4146-bc0d-742d7c1953ff/volumes" Oct 30 01:25:58.235064 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:58.235050 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" path="/var/lib/kubelet/pods/f1751c25-d9a7-4672-8ca5-e1fafb277608/volumes" Oct 30 01:25:58.807048 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:25:58.807010 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:00.568037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:00.567998 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:00.758850 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:26:00.758812 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:26:02.250852 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:02.250814 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:02.724160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:02.724120 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2"] Oct 30 01:26:02.779767 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:02.779728 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk"] Oct 30 01:26:02.915361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:02.915325 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:04.096865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:04.096822 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:05.640297 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:05.640255 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:07.202499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:07.202453 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:08.462841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:08.462779 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:08.590129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:08.590086 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:08.701524 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:08.701484 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:09.066667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:09.066619 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:09.196063 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:09.196025 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:09.750738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:09.750680 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:15.746027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:15.745977 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:15.863253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:15.863203 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:16.035066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:16.034973 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm"] Oct 30 01:26:16.037293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:16.037230 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" containerID="cri-o://9a72503bc7c14cc42078c0f02bf4174a37eda035257e3d85f6087647ac8d257e" gracePeriod=30 Oct 30 01:26:16.091235 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:16.091182 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:17.745392 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:17.745346 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" containerID="cri-o://639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70" gracePeriod=45 Oct 30 01:26:17.761253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:17.761220 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:17.803520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:17.803322 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" containerID="cri-o://9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582" gracePeriod=45 Oct 30 01:26:17.814697 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:17.814417 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": EOF" Oct 30 01:26:17.870834 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:17.870797 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:18.083311 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.082166 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 01:26:18.088695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.088568 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/645b8a00-6e13-426f-9474-a2e36a2b943e-tls-certs\" (UniqueName: \"kubernetes.io/secret/645b8a00-6e13-426f-9474-a2e36a2b943e-tls-certs\") pod \"645b8a00-6e13-426f-9474-a2e36a2b943e\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " Oct 30 01:26:18.088695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.088651 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-home\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-home\") pod \"645b8a00-6e13-426f-9474-a2e36a2b943e\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " Oct 30 01:26:18.088695 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.088690 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-dshm\") pod \"645b8a00-6e13-426f-9474-a2e36a2b943e\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " Oct 30 01:26:18.089062 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.088774 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645b8a00-6e13-426f-9474-a2e36a2b943e-kube-api-access-554qp\" (UniqueName: \"kubernetes.io/projected/645b8a00-6e13-426f-9474-a2e36a2b943e-kube-api-access-554qp\") pod \"645b8a00-6e13-426f-9474-a2e36a2b943e\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " Oct 30 01:26:18.089062 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.088807 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-model-cache\") pod \"645b8a00-6e13-426f-9474-a2e36a2b943e\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " Oct 30 01:26:18.089062 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.088864 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-tmp-dir\") pod \"645b8a00-6e13-426f-9474-a2e36a2b943e\" (UID: \"645b8a00-6e13-426f-9474-a2e36a2b943e\") " Oct 30 01:26:18.089285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.089257 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-home" pod "645b8a00-6e13-426f-9474-a2e36a2b943e" (UID: "645b8a00-6e13-426f-9474-a2e36a2b943e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:18.089699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.089563 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-tmp-dir" pod "645b8a00-6e13-426f-9474-a2e36a2b943e" (UID: "645b8a00-6e13-426f-9474-a2e36a2b943e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:18.089699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.089780 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-model-cache" pod "645b8a00-6e13-426f-9474-a2e36a2b943e" (UID: "645b8a00-6e13-426f-9474-a2e36a2b943e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:18.089699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.090025 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.089699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.090045 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.089699 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.090057 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.092639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.092606 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645b8a00-6e13-426f-9474-a2e36a2b943e-tls-certs" pod "645b8a00-6e13-426f-9474-a2e36a2b943e" (UID: "645b8a00-6e13-426f-9474-a2e36a2b943e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:18.092639 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.092617 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645b8a00-6e13-426f-9474-a2e36a2b943e-kube-api-access-554qp" pod "645b8a00-6e13-426f-9474-a2e36a2b943e" (UID: "645b8a00-6e13-426f-9474-a2e36a2b943e"). InnerVolumeSpecName "kube-api-access-554qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:18.092929 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.092901 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-dshm" pod "645b8a00-6e13-426f-9474-a2e36a2b943e" (UID: "645b8a00-6e13-426f-9474-a2e36a2b943e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:18.114893 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.114859 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 01:26:18.190657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.190596 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-model-cache\") pod \"64047364-1124-4828-ac28-6151d4bdd2f4\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " Oct 30 01:26:18.190904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.190668 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-home\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-home\") pod \"64047364-1124-4828-ac28-6151d4bdd2f4\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " Oct 30 01:26:18.190904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.190765 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-dshm\") pod \"64047364-1124-4828-ac28-6151d4bdd2f4\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " Oct 30 01:26:18.190904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.190850 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-tmp-dir\") pod \"64047364-1124-4828-ac28-6151d4bdd2f4\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " Oct 30 01:26:18.190904 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.190885 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64047364-1124-4828-ac28-6151d4bdd2f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/64047364-1124-4828-ac28-6151d4bdd2f4-tls-certs\") pod \"64047364-1124-4828-ac28-6151d4bdd2f4\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " Oct 30 01:26:18.191151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.190916 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64047364-1124-4828-ac28-6151d4bdd2f4-kube-api-access-v6jq8\" (UniqueName: \"kubernetes.io/projected/64047364-1124-4828-ac28-6151d4bdd2f4-kube-api-access-v6jq8\") pod \"64047364-1124-4828-ac28-6151d4bdd2f4\" (UID: \"64047364-1124-4828-ac28-6151d4bdd2f4\") " Oct 30 01:26:18.191233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.191214 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554qp\" (UniqueName: \"kubernetes.io/projected/645b8a00-6e13-426f-9474-a2e36a2b943e-kube-api-access-554qp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.191293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.191243 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645b8a00-6e13-426f-9474-a2e36a2b943e-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.191293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.191259 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b8a00-6e13-426f-9474-a2e36a2b943e-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.191293 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.191205 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-home" pod "64047364-1124-4828-ac28-6151d4bdd2f4" (UID: "64047364-1124-4828-ac28-6151d4bdd2f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:18.191419 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.191326 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-model-cache" pod "64047364-1124-4828-ac28-6151d4bdd2f4" (UID: "64047364-1124-4828-ac28-6151d4bdd2f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:18.191747 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.191696 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-tmp-dir" pod "64047364-1124-4828-ac28-6151d4bdd2f4" (UID: "64047364-1124-4828-ac28-6151d4bdd2f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:18.193939 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.193910 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64047364-1124-4828-ac28-6151d4bdd2f4-kube-api-access-v6jq8" pod "64047364-1124-4828-ac28-6151d4bdd2f4" (UID: "64047364-1124-4828-ac28-6151d4bdd2f4"). InnerVolumeSpecName "kube-api-access-v6jq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:18.194407 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.194378 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64047364-1124-4828-ac28-6151d4bdd2f4-tls-certs" pod "64047364-1124-4828-ac28-6151d4bdd2f4" (UID: "64047364-1124-4828-ac28-6151d4bdd2f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:18.194726 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.194684 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-dshm" pod "64047364-1124-4828-ac28-6151d4bdd2f4" (UID: "64047364-1124-4828-ac28-6151d4bdd2f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:18.240380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.240335 4517 generic.go:356] "Generic (PLEG): container finished" podID="64047364-1124-4828-ac28-6151d4bdd2f4" containerID="9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582" exitCode=0 Oct 30 01:26:18.240380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.240385 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" Oct 30 01:26:18.240624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.240409 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" event={"ID":"64047364-1124-4828-ac28-6151d4bdd2f4","Type":"ContainerDied","Data":"9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582"} Oct 30 01:26:18.240624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.240460 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk" event={"ID":"64047364-1124-4828-ac28-6151d4bdd2f4","Type":"ContainerDied","Data":"e5f41cd07a812e763f3053e7c7686dfc576e7b4d3a87abc52270eeccb41b4648"} Oct 30 01:26:18.240624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.240483 4517 scope.go:122] "RemoveContainer" containerID="9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582" Oct 30 01:26:18.243107 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.243074 4517 generic.go:356] "Generic (PLEG): container finished" podID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerID="639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70" exitCode=0 Oct 30 01:26:18.243280 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.243216 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" event={"ID":"645b8a00-6e13-426f-9474-a2e36a2b943e","Type":"ContainerDied","Data":"639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70"} Oct 30 01:26:18.243280 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.243258 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" event={"ID":"645b8a00-6e13-426f-9474-a2e36a2b943e","Type":"ContainerDied","Data":"7251b36d77c4247127c8832ddb9b54a4594ab1d5d2182337ee196152ac4b1af2"} Oct 30 01:26:18.243955 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.243931 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2" Oct 30 01:26:18.255787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.255698 4517 scope.go:122] "RemoveContainer" containerID="9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582" Oct 30 01:26:18.256310 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:26:18.256284 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582\": container with ID starting with 9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582 not found: ID does not exist" containerID="9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582" Oct 30 01:26:18.256368 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.256325 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582"} err="failed to get container status \"9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582\": rpc error: code = NotFound desc = could not find container \"9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582\": container with ID starting with 9a0eb7a0b1f7bb09bc6c2ea41d4e2532e33d3d7932955919a43dd70b6b0fa582 not found: ID does not exist" Oct 30 01:26:18.256368 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.256343 4517 scope.go:122] "RemoveContainer" containerID="639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70" Oct 30 01:26:18.271740 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.271695 4517 scope.go:122] "RemoveContainer" containerID="639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70" Oct 30 01:26:18.272633 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:26:18.272594 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70\": container with ID starting with 639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70 not found: ID does not exist" containerID="639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70" Oct 30 01:26:18.272846 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.272648 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70"} err="failed to get container status \"639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70\": rpc error: code = NotFound desc = could not find container \"639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70\": container with ID starting with 639f8d2bd1943d155e8105ec645d2acaaf990e342369255491c3b91fa53a3a70 not found: ID does not exist" Oct 30 01:26:18.292529 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.292463 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.292529 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.292522 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64047364-1124-4828-ac28-6151d4bdd2f4-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.292529 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.292540 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6jq8\" (UniqueName: \"kubernetes.io/projected/64047364-1124-4828-ac28-6151d4bdd2f4-kube-api-access-v6jq8\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.292842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.292555 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.292842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.292569 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.292842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.292580 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64047364-1124-4828-ac28-6151d4bdd2f4-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.305133 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.305092 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk"] Oct 30 01:26:18.309380 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.309347 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/simulator-w5-89a676-kserve-99658d8dc-wf8sk"] Oct 30 01:26:18.339866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.339823 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2"] Oct 30 01:26:18.344694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:18.344659 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-w5-89a676-kserve-759485df45-4x6s2"] Oct 30 01:26:19.533244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:19.533209 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:19.649918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:19.649888 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:20.233634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:20.233588 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" path="/var/lib/kubelet/pods/64047364-1124-4828-ac28-6151d4bdd2f4/volumes" Oct 30 01:26:20.234121 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:20.234098 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" path="/var/lib/kubelet/pods/645b8a00-6e13-426f-9474-a2e36a2b943e/volumes" Oct 30 01:26:21.264181 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.264151 4517 generic.go:356] "Generic (PLEG): container finished" podID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerID="9a72503bc7c14cc42078c0f02bf4174a37eda035257e3d85f6087647ac8d257e" exitCode=0 Oct 30 01:26:21.264668 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.264237 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" event={"ID":"d2479194-1584-4a8f-a67a-bd90fbb3596b","Type":"ContainerDied","Data":"9a72503bc7c14cc42078c0f02bf4174a37eda035257e3d85f6087647ac8d257e"} Oct 30 01:26:21.299202 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.299177 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 01:26:21.322159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322080 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-envoy\") pod \"d2479194-1584-4a8f-a67a-bd90fbb3596b\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " Oct 30 01:26:21.322159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322149 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-socket\") pod \"d2479194-1584-4a8f-a67a-bd90fbb3596b\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " Oct 30 01:26:21.322354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322225 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-token\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-token\") pod \"d2479194-1584-4a8f-a67a-bd90fbb3596b\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " Oct 30 01:26:21.322354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322273 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-credential-socket\") pod \"d2479194-1584-4a8f-a67a-bd90fbb3596b\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " Oct 30 01:26:21.322354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322309 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2479194-1584-4a8f-a67a-bd90fbb3596b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2479194-1584-4a8f-a67a-bd90fbb3596b-istiod-ca-cert\") pod \"d2479194-1584-4a8f-a67a-bd90fbb3596b\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " Oct 30 01:26:21.322521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322351 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-certs\") pod \"d2479194-1584-4a8f-a67a-bd90fbb3596b\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " Oct 30 01:26:21.322521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322413 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-podinfo\") pod \"d2479194-1584-4a8f-a67a-bd90fbb3596b\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " Oct 30 01:26:21.322521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322442 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-kube-api-access-ntvpq\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-kube-api-access-ntvpq\") pod \"d2479194-1584-4a8f-a67a-bd90fbb3596b\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " Oct 30 01:26:21.322521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322459 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-socket" pod "d2479194-1584-4a8f-a67a-bd90fbb3596b" (UID: "d2479194-1584-4a8f-a67a-bd90fbb3596b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:21.322521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322482 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-data\") pod \"d2479194-1584-4a8f-a67a-bd90fbb3596b\" (UID: \"d2479194-1584-4a8f-a67a-bd90fbb3596b\") " Oct 30 01:26:21.322838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322564 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-credential-socket" pod "d2479194-1584-4a8f-a67a-bd90fbb3596b" (UID: "d2479194-1584-4a8f-a67a-bd90fbb3596b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:21.322838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322782 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2479194-1584-4a8f-a67a-bd90fbb3596b-istiod-ca-cert" pod "d2479194-1584-4a8f-a67a-bd90fbb3596b" (UID: "d2479194-1584-4a8f-a67a-bd90fbb3596b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:21.322838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322796 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:21.322838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322799 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-certs" pod "d2479194-1584-4a8f-a67a-bd90fbb3596b" (UID: "d2479194-1584-4a8f-a67a-bd90fbb3596b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:21.322838 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322816 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:21.323151 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.322995 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-data" pod "d2479194-1584-4a8f-a67a-bd90fbb3596b" (UID: "d2479194-1584-4a8f-a67a-bd90fbb3596b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:21.324834 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.324809 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-podinfo" pod "d2479194-1584-4a8f-a67a-bd90fbb3596b" (UID: "d2479194-1584-4a8f-a67a-bd90fbb3596b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:26:21.325016 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.324994 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-kube-api-access-ntvpq" pod "d2479194-1584-4a8f-a67a-bd90fbb3596b" (UID: "d2479194-1584-4a8f-a67a-bd90fbb3596b"). InnerVolumeSpecName "kube-api-access-ntvpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:21.325091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.325075 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-envoy" pod "d2479194-1584-4a8f-a67a-bd90fbb3596b" (UID: "d2479194-1584-4a8f-a67a-bd90fbb3596b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:21.325130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.325092 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-token" pod "d2479194-1584-4a8f-a67a-bd90fbb3596b" (UID: "d2479194-1584-4a8f-a67a-bd90fbb3596b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:21.424083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.424046 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:21.424083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.424075 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntvpq\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-kube-api-access-ntvpq\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:21.424083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.424086 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:21.424335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.424095 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:21.424335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.424103 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2479194-1584-4a8f-a67a-bd90fbb3596b-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:21.424335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.424110 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2479194-1584-4a8f-a67a-bd90fbb3596b-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:21.424335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:21.424140 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2479194-1584-4a8f-a67a-bd90fbb3596b-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:22.273587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:22.273551 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" event={"ID":"d2479194-1584-4a8f-a67a-bd90fbb3596b","Type":"ContainerDied","Data":"7d5904ebca1aef138e403e05a318539aa3ebf8e849340c38fe7bc333ae4e4dce"} Oct 30 01:26:22.274096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:22.273611 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm" Oct 30 01:26:22.274096 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:22.273630 4517 scope.go:122] "RemoveContainer" containerID="9a72503bc7c14cc42078c0f02bf4174a37eda035257e3d85f6087647ac8d257e" Oct 30 01:26:22.330262 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:22.330230 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm"] Oct 30 01:26:22.340551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:22.340518 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-tbhdm"] Oct 30 01:26:24.233159 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:24.233123 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" path="/var/lib/kubelet/pods/d2479194-1584-4a8f-a67a-bd90fbb3596b/volumes" Oct 30 01:26:25.463083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:25.463042 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:25.881756 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:25.881724 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd"] Oct 30 01:26:25.881949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:25.881924 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" podUID="12168cf5-4008-4e5e-9799-fc53b267f3be" containerName="maas-api" containerID="cri-o://9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc" gracePeriod=30 Oct 30 01:26:25.961558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:25.961522 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:26.168633 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.168600 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 01:26:26.270054 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.270015 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-metrics-certs\") pod \"12168cf5-4008-4e5e-9799-fc53b267f3be\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " Oct 30 01:26:26.270299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.270072 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12168cf5-4008-4e5e-9799-fc53b267f3be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12168cf5-4008-4e5e-9799-fc53b267f3be-openshift-service-ca\") pod \"12168cf5-4008-4e5e-9799-fc53b267f3be\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " Oct 30 01:26:26.270299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.270171 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-maas-api-tls\") pod \"12168cf5-4008-4e5e-9799-fc53b267f3be\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " Oct 30 01:26:26.270299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.270194 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12168cf5-4008-4e5e-9799-fc53b267f3be-kube-api-access-8wcdx\" (UniqueName: \"kubernetes.io/projected/12168cf5-4008-4e5e-9799-fc53b267f3be-kube-api-access-8wcdx\") pod \"12168cf5-4008-4e5e-9799-fc53b267f3be\" (UID: \"12168cf5-4008-4e5e-9799-fc53b267f3be\") " Oct 30 01:26:26.270496 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.270472 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12168cf5-4008-4e5e-9799-fc53b267f3be-openshift-service-ca" pod "12168cf5-4008-4e5e-9799-fc53b267f3be" (UID: "12168cf5-4008-4e5e-9799-fc53b267f3be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:26.272501 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.272469 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12168cf5-4008-4e5e-9799-fc53b267f3be-kube-api-access-8wcdx" pod "12168cf5-4008-4e5e-9799-fc53b267f3be" (UID: "12168cf5-4008-4e5e-9799-fc53b267f3be"). InnerVolumeSpecName "kube-api-access-8wcdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:26.272648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.272552 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-maas-api-tls" pod "12168cf5-4008-4e5e-9799-fc53b267f3be" (UID: "12168cf5-4008-4e5e-9799-fc53b267f3be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:26.272648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.272560 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-metrics-certs" pod "12168cf5-4008-4e5e-9799-fc53b267f3be" (UID: "12168cf5-4008-4e5e-9799-fc53b267f3be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:26.304316 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.304281 4517 generic.go:356] "Generic (PLEG): container finished" podID="12168cf5-4008-4e5e-9799-fc53b267f3be" containerID="9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc" exitCode=0 Oct 30 01:26:26.304458 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.304377 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" Oct 30 01:26:26.304499 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.304375 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" event={"ID":"12168cf5-4008-4e5e-9799-fc53b267f3be","Type":"ContainerDied","Data":"9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc"} Oct 30 01:26:26.304541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.304502 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd" event={"ID":"12168cf5-4008-4e5e-9799-fc53b267f3be","Type":"ContainerDied","Data":"e73be9e0fd5483c81fdc3404b87e6692d48e7b0dcc1e0fe0d4f3b5a0c452a413"} Oct 30 01:26:26.304541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.304535 4517 scope.go:122] "RemoveContainer" containerID="9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc" Oct 30 01:26:26.314449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.314430 4517 scope.go:122] "RemoveContainer" containerID="9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc" Oct 30 01:26:26.314697 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:26:26.314674 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc\": container with ID starting with 9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc not found: ID does not exist" containerID="9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc" Oct 30 01:26:26.314788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.314739 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc"} err="failed to get container status \"9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc\": rpc error: code = NotFound desc = could not find container \"9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc\": container with ID starting with 9bd870d29c8666066fe983ef41ddfcd6eec0d57dd4d5ce2641420591430166dc not found: ID does not exist" Oct 30 01:26:26.350132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.350099 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd"] Oct 30 01:26:26.353913 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.353887 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-89a676-5dc854b5bd-85hsd"] Oct 30 01:26:26.371590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.371558 4517 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-metrics-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:26.371590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.371582 4517 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12168cf5-4008-4e5e-9799-fc53b267f3be-openshift-service-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:26.371590 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.371592 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12168cf5-4008-4e5e-9799-fc53b267f3be-maas-api-tls\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:26.371829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.371601 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wcdx\" (UniqueName: \"kubernetes.io/projected/12168cf5-4008-4e5e-9799-fc53b267f3be-kube-api-access-8wcdx\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:26.437802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.437690 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:26.559983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:26.559937 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:27.002244 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:27.002206 4517 scope.go:122] "RemoveContainer" containerID="6955bc015ead17d18d321d727d66850312220da922630ba43a0be15ac003c5b7" Oct 30 01:26:27.011038 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:27.011006 4517 scope.go:122] "RemoveContainer" containerID="19adbce1e28291954e14451fe842be4e19453868a8dbcd6fa646925c27291f9d" Oct 30 01:26:27.018869 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:27.018852 4517 scope.go:122] "RemoveContainer" containerID="0935717c22c97b89fe12b67eb453edc61916acb6473475661fb6921ad8e157d1" Oct 30 01:26:27.026647 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:27.026625 4517 scope.go:122] "RemoveContainer" containerID="ec0a148f978dacd753b8314eb718a6ad3a5f63b773ea73cab1a45be711aa5f60" Oct 30 01:26:27.034254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:27.034238 4517 scope.go:122] "RemoveContainer" containerID="bccefeeeb348be7012d832f6be4db07034172b8d81195e499b9b851f31de76f6" Oct 30 01:26:27.040565 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:27.040531 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:28.236659 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:28.236612 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12168cf5-4008-4e5e-9799-fc53b267f3be" path="/var/lib/kubelet/pods/12168cf5-4008-4e5e-9799-fc53b267f3be/volumes" Oct 30 01:26:30.760447 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:26:30.760411 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:26:33.391681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:33.391641 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:33.504921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:33.504885 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:33.716364 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:33.716279 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l"] Oct 30 01:26:33.716573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:33.716547 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" containerID="cri-o://50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468" gracePeriod=30 Oct 30 01:26:33.724522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:33.724466 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:39.027241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.027208 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.087294 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-podinfo\") pod \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.087352 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-credential-socket\") pod \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.087397 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-token\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-token\") pod \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.087444 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-kube-api-access-r6m7d\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-kube-api-access-r6m7d\") pod \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.087476 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-socket\") pod \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.087508 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-envoy\") pod \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.087548 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istiod-ca-cert\") pod \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.087577 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-certs\") pod \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.087605 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-data\") pod \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\" (UID: \"e3ef060c-8ab7-41a5-935b-8c7fa84559f7\") " Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.088221 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-data" pod "e3ef060c-8ab7-41a5-935b-8c7fa84559f7" (UID: "e3ef060c-8ab7-41a5-935b-8c7fa84559f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.091540 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-credential-socket" pod "e3ef060c-8ab7-41a5-935b-8c7fa84559f7" (UID: "e3ef060c-8ab7-41a5-935b-8c7fa84559f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.091612 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-certs" pod "e3ef060c-8ab7-41a5-935b-8c7fa84559f7" (UID: "e3ef060c-8ab7-41a5-935b-8c7fa84559f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.091792 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istiod-ca-cert" pod "e3ef060c-8ab7-41a5-935b-8c7fa84559f7" (UID: "e3ef060c-8ab7-41a5-935b-8c7fa84559f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:39.093170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.092134 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-socket" pod "e3ef060c-8ab7-41a5-935b-8c7fa84559f7" (UID: "e3ef060c-8ab7-41a5-935b-8c7fa84559f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:39.114100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.113900 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-podinfo" pod "e3ef060c-8ab7-41a5-935b-8c7fa84559f7" (UID: "e3ef060c-8ab7-41a5-935b-8c7fa84559f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:26:39.114616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.114554 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-kube-api-access-r6m7d" pod "e3ef060c-8ab7-41a5-935b-8c7fa84559f7" (UID: "e3ef060c-8ab7-41a5-935b-8c7fa84559f7"). InnerVolumeSpecName "kube-api-access-r6m7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:39.114616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.114555 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-token" pod "e3ef060c-8ab7-41a5-935b-8c7fa84559f7" (UID: "e3ef060c-8ab7-41a5-935b-8c7fa84559f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:39.114616 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.114553 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-envoy" pod "e3ef060c-8ab7-41a5-935b-8c7fa84559f7" (UID: "e3ef060c-8ab7-41a5-935b-8c7fa84559f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:39.188830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.188792 4517 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-podinfo\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.188830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.188825 4517 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-credential-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.188830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.188834 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.189207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.188843 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6m7d\" (UniqueName: \"kubernetes.io/projected/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-kube-api-access-r6m7d\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.189207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.188854 4517 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-socket\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.189207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.188863 4517 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-envoy\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.189207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.188872 4517 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istiod-ca-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.189207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.188885 4517 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-workload-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.189207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.188896 4517 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ef060c-8ab7-41a5-935b-8c7fa84559f7-istio-data\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.371325 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.371280 4517 generic.go:356] "Generic (PLEG): container finished" podID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerID="50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468" exitCode=0 Oct 30 01:26:39.371548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.371345 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" Oct 30 01:26:39.371548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.371380 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" event={"ID":"e3ef060c-8ab7-41a5-935b-8c7fa84559f7","Type":"ContainerDied","Data":"50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468"} Oct 30 01:26:39.371548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.371430 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l" event={"ID":"e3ef060c-8ab7-41a5-935b-8c7fa84559f7","Type":"ContainerDied","Data":"7a4d87bdd6968872c08b2b558dd55442867c96a3ec05826aac758d3c7ff5895b"} Oct 30 01:26:39.371548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.371466 4517 scope.go:122] "RemoveContainer" containerID="50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468" Oct 30 01:26:39.387185 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.387142 4517 scope.go:122] "RemoveContainer" containerID="50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468" Oct 30 01:26:39.387644 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:26:39.387609 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468\": container with ID starting with 50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468 not found: ID does not exist" containerID="50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468" Oct 30 01:26:39.387796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.387658 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468"} err="failed to get container status \"50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468\": rpc error: code = NotFound desc = could not find container \"50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468\": container with ID starting with 50b7495f2f47d58835ef008b87f7d8772dd3ed7e6b2c8374747057d35ccd3468 not found: ID does not exist" Oct 30 01:26:39.420034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.419992 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l"] Oct 30 01:26:39.430386 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:39.430347 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kzw7l"] Oct 30 01:26:40.132571 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:40.132537 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:40.234180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:40.234139 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" path="/var/lib/kubelet/pods/e3ef060c-8ab7-41a5-935b-8c7fa84559f7/volumes" Oct 30 01:26:40.250263 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:40.250203 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:40.629660 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:40.629620 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:46.303127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:46.303091 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:57.797788 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.797744 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2"] Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798221 4517 cpu_manager.go:414] "containerMap: removing container" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798240 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798259 4517 cpu_manager.go:414] "containerMap: removing container" podUID="b10e2a7e-6528-41bf-a775-9fc836a39ba3" containerName="revoke-keys" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798268 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10e2a7e-6528-41bf-a775-9fc836a39ba3" containerName="revoke-keys" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798287 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798295 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798304 4517 cpu_manager.go:414] "containerMap: removing container" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798310 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798320 4517 cpu_manager.go:414] "containerMap: removing container" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798326 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798342 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798349 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798358 4517 cpu_manager.go:414] "containerMap: removing container" podUID="94d6be96-d442-4e00-b51c-6bf15b11a4b6" containerName="payload-processing" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798364 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d6be96-d442-4e00-b51c-6bf15b11a4b6" containerName="payload-processing" Oct 30 01:26:57.798374 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798380 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798388 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798400 4517 cpu_manager.go:414] "containerMap: removing container" podUID="12168cf5-4008-4e5e-9799-fc53b267f3be" containerName="maas-api" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798408 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12168cf5-4008-4e5e-9799-fc53b267f3be" containerName="maas-api" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798420 4517 cpu_manager.go:414] "containerMap: removing container" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798428 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798439 4517 cpu_manager.go:414] "containerMap: removing container" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798447 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798458 4517 cpu_manager.go:414] "containerMap: removing container" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798465 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798476 4517 cpu_manager.go:414] "containerMap: removing container" podUID="072afb9f-694b-40a8-8715-543942df57cf" containerName="payload-pre-processing" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798483 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072afb9f-694b-40a8-8715-543942df57cf" containerName="payload-pre-processing" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798575 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ef060c-8ab7-41a5-935b-8c7fa84559f7" containerName="istio-proxy" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798593 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="12168cf5-4008-4e5e-9799-fc53b267f3be" containerName="maas-api" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798602 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1751c25-d9a7-4672-8ca5-e1fafb277608" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798610 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacc172f-6097-4a9d-bf9e-cbc32a88bf4e" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798618 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="64047364-1124-4828-ac28-6151d4bdd2f4" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798626 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="072afb9f-694b-40a8-8715-543942df57cf" containerName="payload-pre-processing" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798635 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3488a16-3617-4db6-ae61-8c93a76f46e1" containerName="istio-proxy" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798642 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10e2a7e-6528-41bf-a775-9fc836a39ba3" containerName="revoke-keys" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798653 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fabd8b-f688-4146-bc0d-742d7c1953ff" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798661 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="251c373b-a73a-4aef-b678-b579391a6b96" containerName="istio-proxy" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798669 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="645b8a00-6e13-426f-9474-a2e36a2b943e" containerName="main" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798680 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2479194-1584-4a8f-a67a-bd90fbb3596b" containerName="istio-proxy" Oct 30 01:26:57.799227 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.798687 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d6be96-d442-4e00-b51c-6bf15b11a4b6" containerName="payload-processing" Oct 30 01:26:57.802282 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.802258 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:57.829644 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.829608 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2"] Oct 30 01:26:57.956757 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.956664 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:57.957000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.956826 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:57.957000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.956907 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:57.957000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.956945 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:57.957000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.956993 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:57.957203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.957015 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:57.957203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.957038 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wn5l\" (UniqueName: \"kubernetes.io/projected/08f94bdb-35af-4ac9-b852-68a5cd24634f-kube-api-access-8wn5l\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:57.957203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.957137 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:57.957203 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:57.957185 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f94bdb-35af-4ac9-b852-68a5cd24634f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058033 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058083 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058132 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058111 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058137 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058164 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058188 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wn5l\" (UniqueName: \"kubernetes.io/projected/08f94bdb-35af-4ac9-b852-68a5cd24634f-kube-api-access-8wn5l\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058245 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058305 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f94bdb-35af-4ac9-b852-68a5cd24634f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058358 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058548 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058571 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058749 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.058856 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.058792 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.059506 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.059482 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f94bdb-35af-4ac9-b852-68a5cd24634f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.060694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.060677 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.060965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.060946 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.080027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.079990 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f94bdb-35af-4ac9-b852-68a5cd24634f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.080235 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.080209 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wn5l\" (UniqueName: \"kubernetes.io/projected/08f94bdb-35af-4ac9-b852-68a5cd24634f-kube-api-access-8wn5l\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2\" (UID: \"08f94bdb-35af-4ac9-b852-68a5cd24634f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.117013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.116972 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:26:58.481533 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:26:58.481500 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f94bdb_35af_4ac9_b852_68a5cd24634f.slice/crio-a1e8b6aa1475f0e902443ef835d21de8ed34874c0741a24caa8550db97c69d3b WatchSource:0}: Error finding container a1e8b6aa1475f0e902443ef835d21de8ed34874c0741a24caa8550db97c69d3b: Status 404 returned error can't find the container with id a1e8b6aa1475f0e902443ef835d21de8ed34874c0741a24caa8550db97c69d3b Oct 30 01:26:58.483586 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.483552 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:26:58.483667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.483624 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:26:58.483667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.483652 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:26:58.487257 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:58.487232 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2"] Oct 30 01:26:59.488771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:59.488730 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" event={"ID":"08f94bdb-35af-4ac9-b852-68a5cd24634f","Type":"ContainerStarted","Data":"110db162cce5a6c28fed204c4abbe2ed05718ebb08094fd7b16c0b91df16e902"} Oct 30 01:26:59.489321 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:59.488800 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" event={"ID":"08f94bdb-35af-4ac9-b852-68a5cd24634f","Type":"ContainerStarted","Data":"a1e8b6aa1475f0e902443ef835d21de8ed34874c0741a24caa8550db97c69d3b"} Oct 30 01:26:59.551354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:26:59.551296 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" podStartSLOduration=2.551283239 podStartE2EDuration="2.551283239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:59.540516125 +0000 UTC m=+3157.452643266" watchObservedRunningTime="2026-10-30 01:26:59.551283239 +0000 UTC m=+3157.463410380" Oct 30 01:27:00.118086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:00.118043 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:27:00.119614 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:00.119588 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 01:27:00.119761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:00.119644 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" podUID="08f94bdb-35af-4ac9-b852-68a5cd24634f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 01:27:00.773725 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:27:00.773666 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:27:01.118300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:01.118260 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 01:27:01.118534 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:01.118329 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" podUID="08f94bdb-35af-4ac9-b852-68a5cd24634f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 01:27:02.118016 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:02.117971 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 01:27:02.118553 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:02.118036 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" podUID="08f94bdb-35af-4ac9-b852-68a5cd24634f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 01:27:03.118068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:03.118027 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 01:27:03.118520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:03.118091 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" podUID="08f94bdb-35af-4ac9-b852-68a5cd24634f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 01:27:04.117952 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:04.117908 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 01:27:04.118217 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:04.117973 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" podUID="08f94bdb-35af-4ac9-b852-68a5cd24634f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 01:27:05.118242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:05.118200 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 01:27:05.118667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:05.118276 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" podUID="08f94bdb-35af-4ac9-b852-68a5cd24634f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 01:27:07.118093 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:07.118048 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:27:07.118557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:07.118161 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" podUID="08f94bdb-35af-4ac9-b852-68a5cd24634f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:27:08.159960 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:08.159906 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:27:08.160595 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:08.159989 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" podUID="08f94bdb-35af-4ac9-b852-68a5cd24634f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:27:08.817426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:08.817387 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:27:08.817901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:08.817863 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:27:08.818766 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:08.818750 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2" Oct 30 01:27:13.297401 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:13.297360 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:15.360901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:15.360867 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:19.653569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:19.653511 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:27.116254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:27.116216 4517 scope.go:122] "RemoveContainer" containerID="402e940909855b4cdbe8d4787bc9f6e5294645169299a475cc23a60a554196c8" Oct 30 01:27:27.125619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:27.125588 4517 scope.go:122] "RemoveContainer" containerID="4dda7b4b9a3db62cb0765f9174979f9e20e5088a3a36d39122e63c52bc5c9887" Oct 30 01:27:27.135525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:27.135498 4517 scope.go:122] "RemoveContainer" containerID="8a9f3b4da850bf1f702dd205c2584cd01e0e790b6ef7f4dc55492dba0c5735ae" Oct 30 01:27:27.144688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:27.144663 4517 scope.go:122] "RemoveContainer" containerID="03ee718ef0c7b17071fd3dca33b2ca536466799280e8b24b32faa02af735c17b" Oct 30 01:27:30.759450 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:27:30.759402 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:27:35.032271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:35.032174 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l"] Oct 30 01:27:35.042208 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:35.042166 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1927f3f-lwt6l"] Oct 30 01:27:36.233518 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:36.233480 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af096e76-cc77-421f-a309-e238ccc03f62" path="/var/lib/kubelet/pods/af096e76-cc77-421f-a309-e238ccc03f62/volumes" Oct 30 01:27:58.895666 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:27:58.895623 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:00.759075 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:28:00.759037 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:28:07.116298 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:07.116248 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:13.006360 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.006321 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l"] Oct 30 01:28:13.010080 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.010057 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.048281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.048238 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l"] Oct 30 01:28:13.075623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.075584 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.075623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.075627 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.075892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.075651 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.075892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.075748 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.075892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.075805 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.075892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.075827 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.075892 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.075889 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.076104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.075913 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599fs\" (UniqueName: \"kubernetes.io/projected/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-kube-api-access-599fs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.076104 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.075975 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177302 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177334 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177347 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177487 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177531 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177573 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177621 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177612 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177656 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177696 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599fs\" (UniqueName: \"kubernetes.io/projected/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-kube-api-access-599fs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177778 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.177886 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177823 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.178128 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.177981 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.178196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.178149 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.178196 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.178164 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.178337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.178310 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.180205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.180183 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.180205 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.180201 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.199422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.199389 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.199604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.199447 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-599fs\" (UniqueName: \"kubernetes.io/projected/eee3d8a3-129b-4bf7-bb8e-106edb9b69d5-kube-api-access-599fs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l\" (UID: \"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.322020 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.321975 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:13.645596 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.645504 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l"] Oct 30 01:28:13.648398 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:28:13.648365 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee3d8a3_129b_4bf7_bb8e_106edb9b69d5.slice/crio-3f82094ecebc3523b267da25b592480410f7285ffee51ead9f010304117272c1 WatchSource:0}: Error finding container 3f82094ecebc3523b267da25b592480410f7285ffee51ead9f010304117272c1: Status 404 returned error can't find the container with id 3f82094ecebc3523b267da25b592480410f7285ffee51ead9f010304117272c1 Oct 30 01:28:13.650625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.650591 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:28:13.650752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.650663 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:28:13.650752 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.650692 4517 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 01:28:13.857511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.857470 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" event={"ID":"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5","Type":"ContainerStarted","Data":"0ee25301a9c9766c3e0d40637cea8283b243e6afa0e3952dc70a2f1faf2d93df"} Oct 30 01:28:13.857771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.857526 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" event={"ID":"eee3d8a3-129b-4bf7-bb8e-106edb9b69d5","Type":"ContainerStarted","Data":"3f82094ecebc3523b267da25b592480410f7285ffee51ead9f010304117272c1"} Oct 30 01:28:13.900494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:13.900377 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" podStartSLOduration=1.9003577900000002 podStartE2EDuration="1.90035779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:13.89885652 +0000 UTC m=+3231.810983662" watchObservedRunningTime="2026-10-30 01:28:13.90035779 +0000 UTC m=+3231.812484933" Oct 30 01:28:14.323197 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:14.323160 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:14.325005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:14.324975 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 01:28:14.325115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:14.325037 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" podUID="eee3d8a3-129b-4bf7-bb8e-106edb9b69d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 01:28:15.322559 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:15.322520 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 01:28:15.322786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:15.322575 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" podUID="eee3d8a3-129b-4bf7-bb8e-106edb9b69d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 01:28:16.322975 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:16.322944 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 01:28:16.323416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:16.323019 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" podUID="eee3d8a3-129b-4bf7-bb8e-106edb9b69d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 01:28:17.322922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:17.322877 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 01:28:17.323125 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:17.322952 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" podUID="eee3d8a3-129b-4bf7-bb8e-106edb9b69d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 01:28:18.322631 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:18.322591 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 01:28:18.322872 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:18.322650 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" podUID="eee3d8a3-129b-4bf7-bb8e-106edb9b69d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 01:28:19.322791 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:19.322750 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 01:28:19.323246 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:19.322820 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" podUID="eee3d8a3-129b-4bf7-bb8e-106edb9b69d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 01:28:19.780844 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:19.780736 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:20.322931 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:20.322890 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 01:28:20.323473 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:20.322952 4517 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" podUID="eee3d8a3-129b-4bf7-bb8e-106edb9b69d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 01:28:20.555297 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:20.555257 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:20.670792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:20.670683 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:21.844919 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:21.844884 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:21.845465 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:21.845307 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:21.846221 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:21.846196 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" Oct 30 01:28:27.179089 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:27.179056 4517 scope.go:122] "RemoveContainer" containerID="1ec71b374d134ac6dac0cd7db10fedab435981075240a80f143bc926bb582922" Oct 30 01:28:27.187750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:27.187702 4517 scope.go:122] "RemoveContainer" containerID="d1307271bdac0c67319bca3d09989f2e19529eea09eebceae977c8861a7811ae" Oct 30 01:28:30.759675 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:28:30.759633 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:28:43.089879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.089837 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5"] Oct 30 01:28:43.090496 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.090111 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" podUID="a94b14c6-f1c5-433f-9843-9c7aefea72c0" containerName="discovery" containerID="cri-o://0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc" gracePeriod=30 Oct 30 01:28:43.346107 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.346042 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 01:28:43.365292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.365257 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-kubeconfig\") pod \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " Oct 30 01:28:43.365497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.365309 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-ca-configmap\") pod \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " Oct 30 01:28:43.365497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.365337 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94b14c6-f1c5-433f-9843-9c7aefea72c0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94b14c6-f1c5-433f-9843-9c7aefea72c0-local-certs\") pod \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " Oct 30 01:28:43.365497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.365361 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-token\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-token\") pod \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " Oct 30 01:28:43.365497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.365389 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-dns-cert\") pod \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " Oct 30 01:28:43.365497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.365433 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-kube-api-access-csldg\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-kube-api-access-csldg\") pod \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " Oct 30 01:28:43.365497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.365491 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-cacerts\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-cacerts\") pod \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\" (UID: \"a94b14c6-f1c5-433f-9843-9c7aefea72c0\") " Oct 30 01:28:43.365909 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.365881 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-ca-configmap" pod "a94b14c6-f1c5-433f-9843-9c7aefea72c0" (UID: "a94b14c6-f1c5-433f-9843-9c7aefea72c0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:43.367966 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.367922 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94b14c6-f1c5-433f-9843-9c7aefea72c0-local-certs" pod "a94b14c6-f1c5-433f-9843-9c7aefea72c0" (UID: "a94b14c6-f1c5-433f-9843-9c7aefea72c0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:43.368129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.368104 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-cacerts" pod "a94b14c6-f1c5-433f-9843-9c7aefea72c0" (UID: "a94b14c6-f1c5-433f-9843-9c7aefea72c0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:43.368336 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.368308 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-dns-cert" pod "a94b14c6-f1c5-433f-9843-9c7aefea72c0" (UID: "a94b14c6-f1c5-433f-9843-9c7aefea72c0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:43.368420 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.368363 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-token" pod "a94b14c6-f1c5-433f-9843-9c7aefea72c0" (UID: "a94b14c6-f1c5-433f-9843-9c7aefea72c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:43.368598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.368578 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-kubeconfig" pod "a94b14c6-f1c5-433f-9843-9c7aefea72c0" (UID: "a94b14c6-f1c5-433f-9843-9c7aefea72c0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:43.368751 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.368686 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-kube-api-access-csldg" pod "a94b14c6-f1c5-433f-9843-9c7aefea72c0" (UID: "a94b14c6-f1c5-433f-9843-9c7aefea72c0"). InnerVolumeSpecName "kube-api-access-csldg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:43.466950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.466913 4517 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94b14c6-f1c5-433f-9843-9c7aefea72c0-local-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:28:43.466950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.466942 4517 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-token\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:28:43.466950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.466951 4517 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-dns-cert\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:28:43.466950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.466960 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csldg\" (UniqueName: \"kubernetes.io/projected/a94b14c6-f1c5-433f-9843-9c7aefea72c0-kube-api-access-csldg\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:28:43.467251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.466971 4517 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-cacerts\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:28:43.467251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.466980 4517 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-kubeconfig\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:28:43.467251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:43.466989 4517 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a94b14c6-f1c5-433f-9843-9c7aefea72c0-istio-csr-ca-configmap\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:28:44.035830 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.035792 4517 generic.go:356] "Generic (PLEG): container finished" podID="a94b14c6-f1c5-433f-9843-9c7aefea72c0" containerID="0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc" exitCode=0 Oct 30 01:28:44.036023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.035844 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" Oct 30 01:28:44.036023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.035855 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" event={"ID":"a94b14c6-f1c5-433f-9843-9c7aefea72c0","Type":"ContainerDied","Data":"0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc"} Oct 30 01:28:44.036023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.035910 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5" event={"ID":"a94b14c6-f1c5-433f-9843-9c7aefea72c0","Type":"ContainerDied","Data":"9bf0eb7bd47b1e5fcf5e74039c15fcac309241ba1da66a814b373f1bca9f1605"} Oct 30 01:28:44.036023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.035931 4517 scope.go:122] "RemoveContainer" containerID="0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc" Oct 30 01:28:44.047815 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.047796 4517 scope.go:122] "RemoveContainer" containerID="0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc" Oct 30 01:28:44.048109 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:28:44.048087 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc\": container with ID starting with 0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc not found: ID does not exist" containerID="0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc" Oct 30 01:28:44.048174 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.048122 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc"} err="failed to get container status \"0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc\": rpc error: code = NotFound desc = could not find container \"0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc\": container with ID starting with 0529783d4213bfef1846216aaf2ee3dfaeb1bc6e9f94f5cff9377354cff958fc not found: ID does not exist" Oct 30 01:28:44.091925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.091893 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5"] Oct 30 01:28:44.095619 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.095598 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qbkz5"] Oct 30 01:28:44.233677 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:44.233647 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94b14c6-f1c5-433f-9843-9c7aefea72c0" path="/var/lib/kubelet/pods/a94b14c6-f1c5-433f-9843-9c7aefea72c0/volumes" Oct 30 01:28:49.130983 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:49.130949 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:49.243251 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:49.243216 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:56.028164 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:56.028118 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw"] Oct 30 01:28:56.038212 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:56.038185 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edaaa3d-rddcw"] Oct 30 01:28:56.233464 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:28:56.233416 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56f3a12-1267-487b-b597-ec9514c6a26d" path="/var/lib/kubelet/pods/f56f3a12-1267-487b-b597-ec9514c6a26d/volumes" Oct 30 01:29:00.758669 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:29:00.758629 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:29:19.892664 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:19.892630 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:27.209925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:27.209884 4517 scope.go:122] "RemoveContainer" containerID="dd6b71be78940dc81eb4f8eb1495f8bfb1d36d109ad132c945d76d9006defebb" Oct 30 01:29:27.218409 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:27.218386 4517 scope.go:122] "RemoveContainer" containerID="1e3e77201d00fd24c281d5fe0eae16e1df09e011716a4f8aeff7b712ce569178" Oct 30 01:29:30.346570 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:30.346499 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:30.758669 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:29:30.758581 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:29:34.027207 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:34.027167 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn"] Oct 30 01:29:34.037750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:34.037694 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed237c1-h87vn"] Oct 30 01:29:34.233785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:34.233749 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5d9ba8-59b5-4e95-869f-c068941cafc0" path="/var/lib/kubelet/pods/3c5d9ba8-59b5-4e95-869f-c068941cafc0/volumes" Oct 30 01:29:37.888337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:37.887854 4517 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:29:37.888337 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:37.887956 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l" podUID="eee3d8a3-129b-4bf7-bb8e-106edb9b69d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:29:47.800785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:29:47.800748 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:00.830640 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:30:00.830566 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:30:03.089264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.089224 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d"] Oct 30 01:30:03.089812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.089752 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a94b14c6-f1c5-433f-9843-9c7aefea72c0" containerName="discovery" Oct 30 01:30:03.089812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.089768 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94b14c6-f1c5-433f-9843-9c7aefea72c0" containerName="discovery" Oct 30 01:30:03.089929 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.089878 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94b14c6-f1c5-433f-9843-9c7aefea72c0" containerName="discovery" Oct 30 01:30:03.093117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.093092 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.106129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.106096 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:03.116634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.116600 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:30:03.116634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.116624 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:30:03.116926 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.116667 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:30:03.120023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.119990 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d"] Oct 30 01:30:03.169200 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.169154 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wqd6d\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.169433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.169244 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wqd6d\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.169433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.169385 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pjnv\" (UniqueName: \"kubernetes.io/projected/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-kube-api-access-2pjnv\") pod \"maas-api-revoke-sub-79805ca9-wqd6d\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.270604 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.270552 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wqd6d\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.270843 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.270638 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wqd6d\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.270843 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.270738 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjnv\" (UniqueName: \"kubernetes.io/projected/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-kube-api-access-2pjnv\") pod \"maas-api-revoke-sub-79805ca9-wqd6d\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.271321 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.271291 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wqd6d\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.273661 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.273622 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wqd6d\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.292833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.292800 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjnv\" (UniqueName: \"kubernetes.io/projected/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-kube-api-access-2pjnv\") pod \"maas-api-revoke-sub-79805ca9-wqd6d\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.407803 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.407684 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:03.709583 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:03.709481 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d"] Oct 30 01:30:03.714441 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:30:03.714401 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f685d2f_17f6_4ea8_bf06_e466ed2bb3d4.slice/crio-acf9f3056780332ef90c30eb0b91e546023e9b2fc2eaa09639891e533ba2ecb2 WatchSource:0}: Error finding container acf9f3056780332ef90c30eb0b91e546023e9b2fc2eaa09639891e533ba2ecb2: Status 404 returned error can't find the container with id acf9f3056780332ef90c30eb0b91e546023e9b2fc2eaa09639891e533ba2ecb2 Oct 30 01:30:04.471654 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:04.471613 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" event={"ID":"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4","Type":"ContainerStarted","Data":"e0b7f0f8f2b32efa1be3364e163eaf48cbd82855675fd5656d488fe2afb0e2b9"} Oct 30 01:30:04.472150 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:04.471677 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" event={"ID":"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4","Type":"ContainerStarted","Data":"acf9f3056780332ef90c30eb0b91e546023e9b2fc2eaa09639891e533ba2ecb2"} Oct 30 01:30:04.509908 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:04.509853 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" podStartSLOduration=1.5098388360000001 podStartE2EDuration="1.509838836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:04.508671022 +0000 UTC m=+3342.420798162" watchObservedRunningTime="2026-10-30 01:30:04.509838836 +0000 UTC m=+3342.421965978" Oct 30 01:30:05.478488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:05.478449 4517 generic.go:356] "Generic (PLEG): container finished" podID="7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4" containerID="e0b7f0f8f2b32efa1be3364e163eaf48cbd82855675fd5656d488fe2afb0e2b9" exitCode=0 Oct 30 01:30:05.478488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:05.478491 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" event={"ID":"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4","Type":"ContainerDied","Data":"e0b7f0f8f2b32efa1be3364e163eaf48cbd82855675fd5656d488fe2afb0e2b9"} Oct 30 01:30:06.620457 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.620424 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:06.703630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.703586 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-kube-api-access-2pjnv\" (UniqueName: \"kubernetes.io/projected/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-kube-api-access-2pjnv\") pod \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " Oct 30 01:30:06.703873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.703687 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-xks-ca\") pod \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " Oct 30 01:30:06.703873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.703753 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-openshift-ca\") pod \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\" (UID: \"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4\") " Oct 30 01:30:06.704113 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.704083 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-openshift-ca" pod "7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4" (UID: "7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:06.706039 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.706007 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-xks-ca" pod "7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4" (UID: "7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:06.706168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.706077 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-kube-api-access-2pjnv" pod "7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4" (UID: "7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4"). InnerVolumeSpecName "kube-api-access-2pjnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:06.805304 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.805264 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:30:06.805304 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.805300 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:30:06.805304 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:06.805312 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pjnv\" (UniqueName: \"kubernetes.io/projected/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4-kube-api-access-2pjnv\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:30:07.488553 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:07.488511 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" event={"ID":"7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4","Type":"ContainerDied","Data":"acf9f3056780332ef90c30eb0b91e546023e9b2fc2eaa09639891e533ba2ecb2"} Oct 30 01:30:07.488750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:07.488566 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acf9f3056780332ef90c30eb0b91e546023e9b2fc2eaa09639891e533ba2ecb2" Oct 30 01:30:07.488750 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:07.488529 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d" Oct 30 01:30:20.034392 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:20.034349 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:27.244649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:30:27.244609 4517 scope.go:122] "RemoveContainer" containerID="b2b7e92bf98bf5cf6428840f40472683eb91ca6946e22b7afc645e5c0838e14f" Oct 30 01:30:30.760862 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:30:30.760813 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:31:00.758088 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:31:00.758049 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:31:20.172409 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:20.172366 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:27.264786 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:27.264748 4517 scope.go:122] "RemoveContainer" containerID="b00e298927546acc6ed3a38bc9457eb9a35fae5a91859a58191983b21832078b" Oct 30 01:31:30.760937 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:31:30.760896 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:31:36.971335 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:36.971303 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:37.538100 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:37.538058 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:37.658641 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:37.658599 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:41.557906 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:41.557867 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:41.680871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:41.680824 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:43.016388 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:43.016344 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:43.130541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:43.130506 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:43.254606 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:43.254572 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:43.384120 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:43.384083 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:44.780422 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:44.780387 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:46.516748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:46.516689 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:48.223610 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:48.223581 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:49.927285 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:49.927243 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:51.588098 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:51.588064 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:53.447330 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:53.447271 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:55.061675 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:55.061637 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:56.439168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:56.439113 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:56.717403 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:56.717301 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:57.065558 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:57.065523 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:57.257241 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:57.257194 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:58.607547 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:58.607507 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:59.615522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:31:59.615494 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:00.356233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:00.356196 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:00.757761 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:32:00.757650 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:32:02.047630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:02.047578 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:03.824412 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:03.824378 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:05.437690 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:05.437652 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:06.986028 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:06.985998 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:08.719541 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:08.719497 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:10.258007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:10.257974 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:11.898161 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:11.898122 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:13.511447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:13.511413 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:15.120255 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:15.120220 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:16.814907 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:16.814868 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:18.423426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:18.423382 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:20.122928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:20.122887 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:20.263305 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:20.263267 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:21.704218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.704181 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:21.908721 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.908663 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck"] Oct 30 01:32:21.909050 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.909035 4517 cpu_manager.go:414] "containerMap: removing container" podUID="7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4" containerName="revoke-keys" Oct 30 01:32:21.909110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.909051 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4" containerName="revoke-keys" Oct 30 01:32:21.909149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.909116 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4" containerName="revoke-keys" Oct 30 01:32:21.912225 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.912203 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:21.925482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.925455 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:21.935870 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.935839 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:21.936123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.936050 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:21.936742 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.936692 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:32:21.937796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:21.937777 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck"] Oct 30 01:32:22.032451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.032357 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfvw\" (UniqueName: \"kubernetes.io/projected/25f45205-5760-4953-8b48-1cd311a96e1a-kube-api-access-9nfvw\") pod \"maas-api-revoke-sub-f30f9d1f-nddck\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.032451 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.032407 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f30f9d1f-nddck\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.032684 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.032469 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f30f9d1f-nddck\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.133143 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.133114 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfvw\" (UniqueName: \"kubernetes.io/projected/25f45205-5760-4953-8b48-1cd311a96e1a-kube-api-access-9nfvw\") pod \"maas-api-revoke-sub-f30f9d1f-nddck\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.133394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.133159 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f30f9d1f-nddck\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.133394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.133184 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f30f9d1f-nddck\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.133853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.133830 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f30f9d1f-nddck\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.135837 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.135817 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f30f9d1f-nddck\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.155109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.155081 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfvw\" (UniqueName: \"kubernetes.io/projected/25f45205-5760-4953-8b48-1cd311a96e1a-kube-api-access-9nfvw\") pod \"maas-api-revoke-sub-f30f9d1f-nddck\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.224512 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.224471 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:22.524433 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:22.524395 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck"] Oct 30 01:32:22.528699 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:32:22.528668 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f45205_5760_4953_8b48_1cd311a96e1a.slice/crio-b6c651665db1d5553fd6ce4828c935eb7f0ff05df3d0ae9036dbb465d0ed46e5 WatchSource:0}: Error finding container b6c651665db1d5553fd6ce4828c935eb7f0ff05df3d0ae9036dbb465d0ed46e5: Status 404 returned error can't find the container with id b6c651665db1d5553fd6ce4828c935eb7f0ff05df3d0ae9036dbb465d0ed46e5 Oct 30 01:32:23.137392 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:23.137303 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" event={"ID":"25f45205-5760-4953-8b48-1cd311a96e1a","Type":"ContainerStarted","Data":"3d261800a9e479e20d11ec7f56e4effd4cef4557fd74bbc7470d8d0809401310"} Oct 30 01:32:23.137818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:23.137393 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" event={"ID":"25f45205-5760-4953-8b48-1cd311a96e1a","Type":"ContainerStarted","Data":"b6c651665db1d5553fd6ce4828c935eb7f0ff05df3d0ae9036dbb465d0ed46e5"} Oct 30 01:32:23.174568 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:23.174502 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" podStartSLOduration=2.174488409 podStartE2EDuration="2.174488409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:23.173896434 +0000 UTC m=+3481.086023610" watchObservedRunningTime="2026-10-30 01:32:23.174488409 +0000 UTC m=+3481.086615546" Oct 30 01:32:23.362972 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:23.362937 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:24.143275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:24.143240 4517 generic.go:356] "Generic (PLEG): container finished" podID="25f45205-5760-4953-8b48-1cd311a96e1a" containerID="3d261800a9e479e20d11ec7f56e4effd4cef4557fd74bbc7470d8d0809401310" exitCode=0 Oct 30 01:32:24.143735 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:24.143297 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" event={"ID":"25f45205-5760-4953-8b48-1cd311a96e1a","Type":"ContainerDied","Data":"3d261800a9e479e20d11ec7f56e4effd4cef4557fd74bbc7470d8d0809401310"} Oct 30 01:32:25.026652 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.026616 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:25.283372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.283299 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:25.356936 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.356902 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-openshift-ca\") pod \"25f45205-5760-4953-8b48-1cd311a96e1a\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " Oct 30 01:32:25.357126 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.356963 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f45205-5760-4953-8b48-1cd311a96e1a-kube-api-access-9nfvw\" (UniqueName: \"kubernetes.io/projected/25f45205-5760-4953-8b48-1cd311a96e1a-kube-api-access-9nfvw\") pod \"25f45205-5760-4953-8b48-1cd311a96e1a\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " Oct 30 01:32:25.357126 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.357035 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-xks-ca\") pod \"25f45205-5760-4953-8b48-1cd311a96e1a\" (UID: \"25f45205-5760-4953-8b48-1cd311a96e1a\") " Oct 30 01:32:25.357404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.357363 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-openshift-ca" pod "25f45205-5760-4953-8b48-1cd311a96e1a" (UID: "25f45205-5760-4953-8b48-1cd311a96e1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:25.359286 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.359258 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-xks-ca" pod "25f45205-5760-4953-8b48-1cd311a96e1a" (UID: "25f45205-5760-4953-8b48-1cd311a96e1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:25.359384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.359322 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f45205-5760-4953-8b48-1cd311a96e1a-kube-api-access-9nfvw" pod "25f45205-5760-4953-8b48-1cd311a96e1a" (UID: "25f45205-5760-4953-8b48-1cd311a96e1a"). InnerVolumeSpecName "kube-api-access-9nfvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:25.458254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.458218 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:32:25.458254 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.458248 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f45205-5760-4953-8b48-1cd311a96e1a-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:32:25.458481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:25.458265 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nfvw\" (UniqueName: \"kubernetes.io/projected/25f45205-5760-4953-8b48-1cd311a96e1a-kube-api-access-9nfvw\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:32:26.154129 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:26.154091 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" Oct 30 01:32:26.154345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:26.154102 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck" event={"ID":"25f45205-5760-4953-8b48-1cd311a96e1a","Type":"ContainerDied","Data":"b6c651665db1d5553fd6ce4828c935eb7f0ff05df3d0ae9036dbb465d0ed46e5"} Oct 30 01:32:26.154345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:26.154223 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6c651665db1d5553fd6ce4828c935eb7f0ff05df3d0ae9036dbb465d0ed46e5" Oct 30 01:32:26.684404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:26.684371 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:27.285828 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:27.285793 4517 scope.go:122] "RemoveContainer" containerID="a1d9b9f8445043a1af449dadb729050fff3d0cc4554669ff95053407d795afa0" Oct 30 01:32:28.328062 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:28.328013 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:29.965475 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:29.965444 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:30.760048 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:32:30.760010 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:32:31.607083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:31.607031 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:32.913088 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:32.913046 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:33.076319 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.076283 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6"] Oct 30 01:32:33.076759 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.076740 4517 cpu_manager.go:414] "containerMap: removing container" podUID="25f45205-5760-4953-8b48-1cd311a96e1a" containerName="revoke-keys" Oct 30 01:32:33.076869 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.076761 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f45205-5760-4953-8b48-1cd311a96e1a" containerName="revoke-keys" Oct 30 01:32:33.076950 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.076871 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f45205-5760-4953-8b48-1cd311a96e1a" containerName="revoke-keys" Oct 30 01:32:33.081414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.081387 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.097959 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.097926 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:33.105175 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.105140 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:33.105417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.105193 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:33.105417 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.105349 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:32:33.107995 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.107964 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6"] Oct 30 01:32:33.116591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.116551 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.116814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.116603 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.116814 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.116790 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v52qt\" (UniqueName: \"kubernetes.io/projected/da0082fd-d232-4ce1-8ce1-2b896eddd254-kube-api-access-v52qt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.217404 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.217295 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.217625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.217433 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v52qt\" (UniqueName: \"kubernetes.io/projected/da0082fd-d232-4ce1-8ce1-2b896eddd254-kube-api-access-v52qt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.217625 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.217484 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.218210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.218189 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.220146 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.220113 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.228938 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.228903 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:33.239591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.239551 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v52qt\" (UniqueName: \"kubernetes.io/projected/da0082fd-d232-4ce1-8ce1-2b896eddd254-kube-api-access-v52qt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.396965 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.396923 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:33.704882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:33.704837 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6"] Oct 30 01:32:33.708991 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:32:33.708947 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0082fd_d232_4ce1_8ce1_2b896eddd254.slice/crio-5190e4a55c69ff904ff765fc838833e6e548449a5ad0420b9914eaef495de831 WatchSource:0}: Error finding container 5190e4a55c69ff904ff765fc838833e6e548449a5ad0420b9914eaef495de831: Status 404 returned error can't find the container with id 5190e4a55c69ff904ff765fc838833e6e548449a5ad0420b9914eaef495de831 Oct 30 01:32:34.202375 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:34.202336 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" event={"ID":"da0082fd-d232-4ce1-8ce1-2b896eddd254","Type":"ContainerStarted","Data":"02c76117ad9781eb6d9c6ceb3492d747e28271c7d6bb455deaa3f95e8148ac3b"} Oct 30 01:32:34.202833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:34.202398 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" event={"ID":"da0082fd-d232-4ce1-8ce1-2b896eddd254","Type":"ContainerStarted","Data":"5190e4a55c69ff904ff765fc838833e6e548449a5ad0420b9914eaef495de831"} Oct 30 01:32:34.239272 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:34.239147 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" podStartSLOduration=1.239131406 podStartE2EDuration="1.239131406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:34.238336112 +0000 UTC m=+3492.150463253" watchObservedRunningTime="2026-10-30 01:32:34.239131406 +0000 UTC m=+3492.151258546" Oct 30 01:32:35.066396 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:35.066359 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:35.210157 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:35.210123 4517 generic.go:356] "Generic (PLEG): container finished" podID="da0082fd-d232-4ce1-8ce1-2b896eddd254" containerID="02c76117ad9781eb6d9c6ceb3492d747e28271c7d6bb455deaa3f95e8148ac3b" exitCode=0 Oct 30 01:32:35.210649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:35.210217 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" event={"ID":"da0082fd-d232-4ce1-8ce1-2b896eddd254","Type":"ContainerDied","Data":"02c76117ad9781eb6d9c6ceb3492d747e28271c7d6bb455deaa3f95e8148ac3b"} Oct 30 01:32:36.354517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.354488 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:36.446882 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.446841 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-openshift-ca\") pod \"da0082fd-d232-4ce1-8ce1-2b896eddd254\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " Oct 30 01:32:36.447106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.446909 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-xks-ca\") pod \"da0082fd-d232-4ce1-8ce1-2b896eddd254\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " Oct 30 01:32:36.447106 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.446945 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0082fd-d232-4ce1-8ce1-2b896eddd254-kube-api-access-v52qt\" (UniqueName: \"kubernetes.io/projected/da0082fd-d232-4ce1-8ce1-2b896eddd254-kube-api-access-v52qt\") pod \"da0082fd-d232-4ce1-8ce1-2b896eddd254\" (UID: \"da0082fd-d232-4ce1-8ce1-2b896eddd254\") " Oct 30 01:32:36.447274 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.447248 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-openshift-ca" pod "da0082fd-d232-4ce1-8ce1-2b896eddd254" (UID: "da0082fd-d232-4ce1-8ce1-2b896eddd254"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:36.449201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.449170 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0082fd-d232-4ce1-8ce1-2b896eddd254-kube-api-access-v52qt" pod "da0082fd-d232-4ce1-8ce1-2b896eddd254" (UID: "da0082fd-d232-4ce1-8ce1-2b896eddd254"). InnerVolumeSpecName "kube-api-access-v52qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:36.449330 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.449256 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-xks-ca" pod "da0082fd-d232-4ce1-8ce1-2b896eddd254" (UID: "da0082fd-d232-4ce1-8ce1-2b896eddd254"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:36.547764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.547694 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:32:36.547764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.547762 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0082fd-d232-4ce1-8ce1-2b896eddd254-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:32:36.547994 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.547775 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v52qt\" (UniqueName: \"kubernetes.io/projected/da0082fd-d232-4ce1-8ce1-2b896eddd254-kube-api-access-v52qt\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:32:36.640371 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:36.640334 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:37.221317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:37.221277 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" Oct 30 01:32:37.221317 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:37.221300 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6" event={"ID":"da0082fd-d232-4ce1-8ce1-2b896eddd254","Type":"ContainerDied","Data":"5190e4a55c69ff904ff765fc838833e6e548449a5ad0420b9914eaef495de831"} Oct 30 01:32:37.221552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:37.221374 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5190e4a55c69ff904ff765fc838833e6e548449a5ad0420b9914eaef495de831" Oct 30 01:32:37.952592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:37.952554 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:38.427599 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:38.427563 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:39.956168 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:39.956126 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:41.490732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:41.490678 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:43.217187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:43.217155 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:43.642325 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:43.642279 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:43.790987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:43.790925 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:44.835091 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:44.835057 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:46.249580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:46.249546 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:46.872676 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:46.872641 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:46.982528 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:46.982493 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:46.988299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:46.988262 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6"] Oct 30 01:32:46.998613 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:46.998578 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ff1bf5-dj2g6"] Oct 30 01:32:47.480131 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:47.480095 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:48.233646 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:48.233607 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0082fd-d232-4ce1-8ce1-2b896eddd254" path="/var/lib/kubelet/pods/da0082fd-d232-4ce1-8ce1-2b896eddd254/volumes" Oct 30 01:32:53.540035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:53.539998 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:53.660688 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:53.660648 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:53.953413 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:32:53.953317 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:00.760040 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:33:00.759990 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:33:14.713013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:14.712972 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:15.779275 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:15.779239 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:19.241497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.241454 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6"] Oct 30 01:33:19.242029 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.241726 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" podUID="d95695ed-5cf2-4464-b7ca-b3f7481b6a1d" containerName="manager" containerID="cri-o://ab0e255649131f75c6bfb64365955fc88dcdacc06256ff2b92a4434635d0cbb9" gracePeriod=10 Oct 30 01:33:19.429216 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.429181 4517 generic.go:356] "Generic (PLEG): container finished" podID="d95695ed-5cf2-4464-b7ca-b3f7481b6a1d" containerID="ab0e255649131f75c6bfb64365955fc88dcdacc06256ff2b92a4434635d0cbb9" exitCode=0 Oct 30 01:33:19.429216 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.429225 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" event={"ID":"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d","Type":"ContainerDied","Data":"ab0e255649131f75c6bfb64365955fc88dcdacc06256ff2b92a4434635d0cbb9"} Oct 30 01:33:19.508247 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.508213 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 01:33:19.596092 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.596055 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-kube-api-access-bvkgw\" (UniqueName: \"kubernetes.io/projected/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-kube-api-access-bvkgw\") pod \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\" (UID: \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\") " Oct 30 01:33:19.596359 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.596103 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-extensions-socket-volume\") pod \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\" (UID: \"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d\") " Oct 30 01:33:19.596600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.596571 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-extensions-socket-volume" pod "d95695ed-5cf2-4464-b7ca-b3f7481b6a1d" (UID: "d95695ed-5cf2-4464-b7ca-b3f7481b6a1d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:19.598471 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.598436 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-kube-api-access-bvkgw" pod "d95695ed-5cf2-4464-b7ca-b3f7481b6a1d" (UID: "d95695ed-5cf2-4464-b7ca-b3f7481b6a1d"). InnerVolumeSpecName "kube-api-access-bvkgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:19.697338 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.697297 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvkgw\" (UniqueName: \"kubernetes.io/projected/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-kube-api-access-bvkgw\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:33:19.697338 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:19.697333 4517 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d-extensions-socket-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:33:20.387555 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:20.387520 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:20.435366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:20.435323 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" event={"ID":"d95695ed-5cf2-4464-b7ca-b3f7481b6a1d","Type":"ContainerDied","Data":"4664146a72eef8a5dbc8268552f1a3e4ca1bd32d35511c51d25d8da00e1b20b2"} Oct 30 01:33:20.435366 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:20.435361 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6" Oct 30 01:33:20.435623 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:20.435405 4517 scope.go:122] "RemoveContainer" containerID="ab0e255649131f75c6bfb64365955fc88dcdacc06256ff2b92a4434635d0cbb9" Oct 30 01:33:20.494683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:20.494599 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6"] Oct 30 01:33:20.505649 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:20.505608 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-gz9r6"] Oct 30 01:33:21.446522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:21.446488 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:22.239624 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:22.234556 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95695ed-5cf2-4464-b7ca-b3f7481b6a1d" path="/var/lib/kubelet/pods/d95695ed-5cf2-4464-b7ca-b3f7481b6a1d/volumes" Oct 30 01:33:30.759405 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:33:30.759361 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:33:40.519511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:40.519453 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:40.645156 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:33:40.645124 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:00.758816 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:34:00.758774 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:34:20.524179 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:34:20.524140 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:30.759352 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:34:30.759303 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:35:00.765085 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:35:00.765033 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:35:07.037736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:07.037637 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d"] Oct 30 01:35:07.048079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:07.048049 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wqd6d"] Oct 30 01:35:08.233144 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:08.233098 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4" path="/var/lib/kubelet/pods/7f685d2f-17f6-4ea8-bf06-e466ed2bb3d4/volumes" Oct 30 01:35:11.003384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:11.003349 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:11.857147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:11.857113 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:15.546511 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:15.546473 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:15.672001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:15.671961 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:17.080284 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.080253 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:17.211172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.211135 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:17.344800 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.344689 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:17.392289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.392252 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6"] Oct 30 01:35:17.392594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.392581 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d95695ed-5cf2-4464-b7ca-b3f7481b6a1d" containerName="manager" Oct 30 01:35:17.392594 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.392594 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95695ed-5cf2-4464-b7ca-b3f7481b6a1d" containerName="manager" Oct 30 01:35:17.392691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.392607 4517 cpu_manager.go:414] "containerMap: removing container" podUID="da0082fd-d232-4ce1-8ce1-2b896eddd254" containerName="revoke-keys" Oct 30 01:35:17.392691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.392612 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0082fd-d232-4ce1-8ce1-2b896eddd254" containerName="revoke-keys" Oct 30 01:35:17.392691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.392668 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95695ed-5cf2-4464-b7ca-b3f7481b6a1d" containerName="manager" Oct 30 01:35:17.392691 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.392678 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0082fd-d232-4ce1-8ce1-2b896eddd254" containerName="revoke-keys" Oct 30 01:35:17.395726 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.395686 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:17.420640 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.420599 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ca8c2a29-dockercfg-hkh2x\"" Oct 30 01:35:17.422372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.422342 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6"] Oct 30 01:35:17.464301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.464267 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7e040d\"" Oct 30 01:35:17.474764 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.474721 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6\" (UID: \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:17.474986 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.474775 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-kube-api-access-66c2h\") pod \"payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6\" (UID: \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:17.575802 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.575751 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-kube-api-access-66c2h\") pod \"payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6\" (UID: \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:17.576027 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.575916 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6\" (UID: \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:17.576629 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.576599 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6\" (UID: \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:17.597427 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.597367 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-kube-api-access-66c2h\") pod \"payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6\" (UID: \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:17.713835 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:17.713792 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:18.016543 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:18.016448 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6"] Oct 30 01:35:18.021102 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:35:18.021072 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bbc5f2_3224_4415_96d4_5f120ff7cfc3.slice/crio-5b028b583f392e4c3ea7c2579c330e471ad04afc82a1e435cb509eb7f48437bb WatchSource:0}: Error finding container 5b028b583f392e4c3ea7c2579c330e471ad04afc82a1e435cb509eb7f48437bb: Status 404 returned error can't find the container with id 5b028b583f392e4c3ea7c2579c330e471ad04afc82a1e435cb509eb7f48437bb Oct 30 01:35:18.872916 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:18.872873 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:18.987397 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:18.987358 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" event={"ID":"39bbc5f2-3224-4415-96d4-5f120ff7cfc3","Type":"ContainerStarted","Data":"94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad"} Oct 30 01:35:18.987657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:18.987413 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" event={"ID":"39bbc5f2-3224-4415-96d4-5f120ff7cfc3","Type":"ContainerStarted","Data":"5b028b583f392e4c3ea7c2579c330e471ad04afc82a1e435cb509eb7f48437bb"} Oct 30 01:35:18.987657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:18.987494 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:19.025526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:19.025458 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" podStartSLOduration=2.025438495 podStartE2EDuration="2.025438495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:19.023965034 +0000 UTC m=+3656.936092178" watchObservedRunningTime="2026-10-30 01:35:19.025438495 +0000 UTC m=+3656.937565635" Oct 30 01:35:20.534498 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:20.534467 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:20.663609 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:20.663574 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:22.319447 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:22.319398 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:23.879055 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:23.879005 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:25.539391 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:25.539359 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:27.404242 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:27.404209 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:29.024732 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:29.024689 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:29.994551 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:29.994520 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:35:30.450638 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:30.450603 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:30.729345 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:30.729261 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:30.758679 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:35:30.758641 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:35:31.037956 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:31.037922 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:31.239007 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:31.238962 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:31.366180 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:31.366088 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:31.555777 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:31.555742 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:31.684270 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:31.684185 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:32.540970 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.540933 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:32.576622 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.576591 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt"] Oct 30 01:35:32.580019 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.579993 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.598600 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.598568 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7e040d\"/\"kube-root-ca.crt\"" Oct 30 01:35:32.606256 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.606232 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7e040d\"/\"openshift-service-ca.crt\"" Oct 30 01:35:32.606256 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.606252 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7e040d\"/\"default-dockercfg-nn2vf\"" Oct 30 01:35:32.606588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.606569 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7e040d\"/\"simulator-single-main-7e040d-kserve-self-signed-certs\"" Oct 30 01:35:32.610569 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.610537 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt"] Oct 30 01:35:32.660685 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.660650 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:32.710071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.710035 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-home\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.710071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.710075 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-tmp-dir\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.710376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.710108 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e467827d-8014-468c-a4e3-df6448b8ff15-tls-certs\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.710376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.710221 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-dshm\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.710376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.710305 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntxp\" (UniqueName: \"kubernetes.io/projected/e467827d-8014-468c-a4e3-df6448b8ff15-kube-api-access-7ntxp\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.710376 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.710330 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-model-cache\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.811822 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.811701 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-model-cache\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.812026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.811822 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-home\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.812026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.811863 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-tmp-dir\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.812026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.811918 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e467827d-8014-468c-a4e3-df6448b8ff15-tls-certs\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.812026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.811953 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-dshm\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.812026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.812018 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntxp\" (UniqueName: \"kubernetes.io/projected/e467827d-8014-468c-a4e3-df6448b8ff15-kube-api-access-7ntxp\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.812344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.812245 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-model-cache\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.812344 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.812281 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-tmp-dir\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.812669 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.812644 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-home\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.815299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.815269 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-dshm\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.816066 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.816031 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e467827d-8014-468c-a4e3-df6448b8ff15-tls-certs\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.834289 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.834240 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntxp\" (UniqueName: \"kubernetes.io/projected/e467827d-8014-468c-a4e3-df6448b8ff15-kube-api-access-7ntxp\") pod \"simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:32.894059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:32.894020 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:33.196650 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:33.196561 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt"] Oct 30 01:35:33.200480 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:35:33.200445 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode467827d_8014_468c_a4e3_df6448b8ff15.slice/crio-b775c868687fc95915e9da498560ea790b334ab11150947bc0b0e5c10936393c WatchSource:0}: Error finding container b775c868687fc95915e9da498560ea790b334ab11150947bc0b0e5c10936393c: Status 404 returned error can't find the container with id b775c868687fc95915e9da498560ea790b334ab11150947bc0b0e5c10936393c Oct 30 01:35:34.051110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:34.051073 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" event={"ID":"e467827d-8014-468c-a4e3-df6448b8ff15","Type":"ContainerStarted","Data":"8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3"} Oct 30 01:35:34.051540 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:34.051122 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" event={"ID":"e467827d-8014-468c-a4e3-df6448b8ff15","Type":"ContainerStarted","Data":"b775c868687fc95915e9da498560ea790b334ab11150947bc0b0e5c10936393c"} Oct 30 01:35:34.089301 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:34.089245 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" podStartSLOduration=2.089231419 podStartE2EDuration="2.089231419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:34.088596748 +0000 UTC m=+3672.000723886" watchObservedRunningTime="2026-10-30 01:35:34.089231419 +0000 UTC m=+3672.001358560" Oct 30 01:35:34.261015 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:34.260981 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:34.894727 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:34.894683 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:36.037736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:36.037658 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:37.665736 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:37.665668 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:39.343865 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:39.343831 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:40.988532 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:40.988497 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:42.646233 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:42.646197 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:42.894928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:42.894895 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:42.907921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:42.907826 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:43.103780 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:43.103745 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:35:44.218517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:44.218464 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:45.787016 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:45.786976 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:47.568311 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:47.568275 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:49.242079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:49.242040 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:51.028061 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:51.028027 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:52.051481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.051439 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8"] Oct 30 01:35:52.055292 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.055269 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.078488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.078455 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:35:52.078744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.078505 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:35:52.078841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.078536 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:35:52.078841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.078598 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:35:52.080186 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.080161 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8"] Oct 30 01:35:52.192833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.192801 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab6d1875-zrnl8\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.193037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.192868 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jp2\" (UniqueName: \"kubernetes.io/projected/a38d1807-0afc-40ba-89c8-a560181367ce-kube-api-access-l6jp2\") pod \"maas-api-revoke-sub-ab6d1875-zrnl8\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.193037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.192943 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab6d1875-zrnl8\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.293792 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.293750 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab6d1875-zrnl8\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.294001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.293914 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jp2\" (UniqueName: \"kubernetes.io/projected/a38d1807-0afc-40ba-89c8-a560181367ce-kube-api-access-l6jp2\") pod \"maas-api-revoke-sub-ab6d1875-zrnl8\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.294001 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.293953 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab6d1875-zrnl8\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.294528 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.294507 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab6d1875-zrnl8\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.296497 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.296480 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab6d1875-zrnl8\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.316603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.316538 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jp2\" (UniqueName: \"kubernetes.io/projected/a38d1807-0afc-40ba-89c8-a560181367ce-kube-api-access-l6jp2\") pod \"maas-api-revoke-sub-ab6d1875-zrnl8\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.368535 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.368492 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:52.623500 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.623393 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:52.667508 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:52.667471 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8"] Oct 30 01:35:52.671701 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:35:52.671673 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38d1807_0afc_40ba_89c8_a560181367ce.slice/crio-d4dcb4bcd12241da64698e79bd7cdd4ae172b70093d67aa29df061915fe8b0f3 WatchSource:0}: Error finding container d4dcb4bcd12241da64698e79bd7cdd4ae172b70093d67aa29df061915fe8b0f3: Status 404 returned error can't find the container with id d4dcb4bcd12241da64698e79bd7cdd4ae172b70093d67aa29df061915fe8b0f3 Oct 30 01:35:53.140357 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:53.140313 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" event={"ID":"a38d1807-0afc-40ba-89c8-a560181367ce","Type":"ContainerStarted","Data":"17f3290d0e617de4e15ef87ebcc221f532716f3361a155e0b618cc34a6725506"} Oct 30 01:35:53.140794 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:53.140407 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" event={"ID":"a38d1807-0afc-40ba-89c8-a560181367ce","Type":"ContainerStarted","Data":"d4dcb4bcd12241da64698e79bd7cdd4ae172b70093d67aa29df061915fe8b0f3"} Oct 30 01:35:53.178165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:53.178110 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" podStartSLOduration=1.178094214 podStartE2EDuration="1.178094214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:53.176233088 +0000 UTC m=+3691.088360232" watchObservedRunningTime="2026-10-30 01:35:53.178094214 +0000 UTC m=+3691.090221389" Oct 30 01:35:54.146079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:54.146040 4517 generic.go:356] "Generic (PLEG): container finished" podID="a38d1807-0afc-40ba-89c8-a560181367ce" containerID="17f3290d0e617de4e15ef87ebcc221f532716f3361a155e0b618cc34a6725506" exitCode=0 Oct 30 01:35:54.146482 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:54.146106 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" event={"ID":"a38d1807-0afc-40ba-89c8-a560181367ce","Type":"ContainerDied","Data":"17f3290d0e617de4e15ef87ebcc221f532716f3361a155e0b618cc34a6725506"} Oct 30 01:35:54.348990 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:54.348953 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:55.288841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.288816 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:55.426218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.426106 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-xks-ca\") pod \"a38d1807-0afc-40ba-89c8-a560181367ce\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " Oct 30 01:35:55.426416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.426235 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38d1807-0afc-40ba-89c8-a560181367ce-kube-api-access-l6jp2\" (UniqueName: \"kubernetes.io/projected/a38d1807-0afc-40ba-89c8-a560181367ce-kube-api-access-l6jp2\") pod \"a38d1807-0afc-40ba-89c8-a560181367ce\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " Oct 30 01:35:55.426416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.426284 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-openshift-ca\") pod \"a38d1807-0afc-40ba-89c8-a560181367ce\" (UID: \"a38d1807-0afc-40ba-89c8-a560181367ce\") " Oct 30 01:35:55.426692 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.426657 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-openshift-ca" pod "a38d1807-0afc-40ba-89c8-a560181367ce" (UID: "a38d1807-0afc-40ba-89c8-a560181367ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:55.428517 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.428491 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-xks-ca" pod "a38d1807-0afc-40ba-89c8-a560181367ce" (UID: "a38d1807-0afc-40ba-89c8-a560181367ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:55.428603 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.428549 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38d1807-0afc-40ba-89c8-a560181367ce-kube-api-access-l6jp2" pod "a38d1807-0afc-40ba-89c8-a560181367ce" (UID: "a38d1807-0afc-40ba-89c8-a560181367ce"). InnerVolumeSpecName "kube-api-access-l6jp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:55.527117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.527081 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6jp2\" (UniqueName: \"kubernetes.io/projected/a38d1807-0afc-40ba-89c8-a560181367ce-kube-api-access-l6jp2\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:35:55.527117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.527112 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:35:55.527117 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.527121 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38d1807-0afc-40ba-89c8-a560181367ce-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:35:55.911271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:55.911221 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:56.158866 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:56.158835 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" Oct 30 01:35:56.159078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:56.158836 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8" event={"ID":"a38d1807-0afc-40ba-89c8-a560181367ce","Type":"ContainerDied","Data":"d4dcb4bcd12241da64698e79bd7cdd4ae172b70093d67aa29df061915fe8b0f3"} Oct 30 01:35:56.159078 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:56.158984 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4dcb4bcd12241da64698e79bd7cdd4ae172b70093d67aa29df061915fe8b0f3" Oct 30 01:35:57.547214 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:57.547176 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:59.170918 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:35:59.170876 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:00.709544 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:00.709512 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:00.761623 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:36:00.761582 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:36:02.419195 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:02.419157 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:02.946767 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:02.946732 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:03.088264 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.088232 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x"] Oct 30 01:36:03.088585 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.088574 4517 cpu_manager.go:414] "containerMap: removing container" podUID="a38d1807-0afc-40ba-89c8-a560181367ce" containerName="revoke-keys" Oct 30 01:36:03.088636 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.088587 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38d1807-0afc-40ba-89c8-a560181367ce" containerName="revoke-keys" Oct 30 01:36:03.088683 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.088648 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38d1807-0afc-40ba-89c8-a560181367ce" containerName="revoke-keys" Oct 30 01:36:03.091287 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.091269 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.114839 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.114806 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:36:03.115023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.114811 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:36:03.115023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.114887 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:36:03.115023 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.114889 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:36:03.116573 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.116550 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x"] Oct 30 01:36:03.193034 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.193002 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5gc\" (UniqueName: \"kubernetes.io/projected/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-kube-api-access-hx5gc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.193226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.193099 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.193226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.193126 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.294577 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.294537 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.294817 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.294672 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5gc\" (UniqueName: \"kubernetes.io/projected/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-kube-api-access-hx5gc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.294876 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.294813 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.295199 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.295181 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.297395 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.297372 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.316761 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.316699 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5gc\" (UniqueName: \"kubernetes.io/projected/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-kube-api-access-hx5gc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.407187 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.407130 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:03.706591 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:03.706452 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x"] Oct 30 01:36:03.711821 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:36:03.711783 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b2442c_fd44_4e42_84f2_2e0f0172dd95.slice/crio-736e5e13433406a54c26ef903f171dd3dae97abb1dd767c9e4795324a98810d6 WatchSource:0}: Error finding container 736e5e13433406a54c26ef903f171dd3dae97abb1dd767c9e4795324a98810d6: Status 404 returned error can't find the container with id 736e5e13433406a54c26ef903f171dd3dae97abb1dd767c9e4795324a98810d6 Oct 30 01:36:04.197110 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:04.197070 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" event={"ID":"d4b2442c-fd44-4e42-84f2-2e0f0172dd95","Type":"ContainerStarted","Data":"5abbc4df44dd24b6cd44f2986aa979cf31cab96fcf7b842ab241232342c0d7f0"} Oct 30 01:36:04.197324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:04.197126 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" event={"ID":"d4b2442c-fd44-4e42-84f2-2e0f0172dd95","Type":"ContainerStarted","Data":"736e5e13433406a54c26ef903f171dd3dae97abb1dd767c9e4795324a98810d6"} Oct 30 01:36:04.268354 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:04.268214 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" podStartSLOduration=1.268198568 podStartE2EDuration="1.268198568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:04.263924591 +0000 UTC m=+3702.176051733" watchObservedRunningTime="2026-10-30 01:36:04.268198568 +0000 UTC m=+3702.180325709" Oct 30 01:36:04.414099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:04.414065 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:05.203005 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:05.202966 4517 generic.go:356] "Generic (PLEG): container finished" podID="d4b2442c-fd44-4e42-84f2-2e0f0172dd95" containerID="5abbc4df44dd24b6cd44f2986aa979cf31cab96fcf7b842ab241232342c0d7f0" exitCode=0 Oct 30 01:36:05.203459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:05.203072 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" event={"ID":"d4b2442c-fd44-4e42-84f2-2e0f0172dd95","Type":"ContainerDied","Data":"5abbc4df44dd24b6cd44f2986aa979cf31cab96fcf7b842ab241232342c0d7f0"} Oct 30 01:36:05.960959 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:05.960917 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:06.344701 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.344672 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:06.428371 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.428338 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-xks-ca\") pod \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " Oct 30 01:36:06.428588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.428404 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-openshift-ca\") pod \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " Oct 30 01:36:06.428588 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.428488 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-kube-api-access-hx5gc\" (UniqueName: \"kubernetes.io/projected/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-kube-api-access-hx5gc\") pod \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\" (UID: \"d4b2442c-fd44-4e42-84f2-2e0f0172dd95\") " Oct 30 01:36:06.428829 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.428804 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-openshift-ca" pod "d4b2442c-fd44-4e42-84f2-2e0f0172dd95" (UID: "d4b2442c-fd44-4e42-84f2-2e0f0172dd95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:06.430642 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.430619 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-xks-ca" pod "d4b2442c-fd44-4e42-84f2-2e0f0172dd95" (UID: "d4b2442c-fd44-4e42-84f2-2e0f0172dd95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:06.430815 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.430793 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-kube-api-access-hx5gc" pod "d4b2442c-fd44-4e42-84f2-2e0f0172dd95" (UID: "d4b2442c-fd44-4e42-84f2-2e0f0172dd95"). InnerVolumeSpecName "kube-api-access-hx5gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:06.530177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.530091 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:06.530177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.530121 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:06.530177 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:06.530131 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx5gc\" (UniqueName: \"kubernetes.io/projected/d4b2442c-fd44-4e42-84f2-2e0f0172dd95-kube-api-access-hx5gc\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:07.213876 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:07.213839 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" Oct 30 01:36:07.213876 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:07.213860 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x" event={"ID":"d4b2442c-fd44-4e42-84f2-2e0f0172dd95","Type":"ContainerDied","Data":"736e5e13433406a54c26ef903f171dd3dae97abb1dd767c9e4795324a98810d6"} Oct 30 01:36:07.214099 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:07.213906 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="736e5e13433406a54c26ef903f171dd3dae97abb1dd767c9e4795324a98810d6" Oct 30 01:36:07.531372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:07.531291 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:07.898004 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:07.897959 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt"] Oct 30 01:36:07.998035 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:07.997999 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:09.234747 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:09.234685 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:10.829372 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:10.829342 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:12.444149 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:12.444105 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:13.539459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:13.539421 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6"] Oct 30 01:36:13.714232 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:13.714193 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:13.859101 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:13.859003 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:14.871537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:14.871500 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:14.989321 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:14.989288 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x"] Oct 30 01:36:14.989522 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:14.989320 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:14.999387 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:14.999352 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7e040d-l4m6x"] Oct 30 01:36:15.467951 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:15.467918 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:16.233584 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:16.233540 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b2442c-fd44-4e42-84f2-2e0f0172dd95" path="/var/lib/kubelet/pods/d4b2442c-fd44-4e42-84f2-2e0f0172dd95/volumes" Oct 30 01:36:18.540350 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:18.540303 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" podUID="39bbc5f2-3224-4415-96d4-5f120ff7cfc3" containerName="payload-pre-processing" containerID="cri-o://94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad" gracePeriod=25 Oct 30 01:36:19.993755 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:19.993695 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" podUID="39bbc5f2-3224-4415-96d4-5f120ff7cfc3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:36:20.779645 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:20.779611 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:21.343807 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:21.343771 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:21.573945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:21.573915 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:21.707394 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:21.707301 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:22.919300 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:22.919258 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" podUID="e467827d-8014-468c-a4e3-df6448b8ff15" containerName="main" containerID="cri-o://8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3" gracePeriod=45 Oct 30 01:36:23.174200 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.174125 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:36:23.287191 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287160 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e467827d-8014-468c-a4e3-df6448b8ff15-tls-certs\" (UniqueName: \"kubernetes.io/secret/e467827d-8014-468c-a4e3-df6448b8ff15-tls-certs\") pod \"e467827d-8014-468c-a4e3-df6448b8ff15\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " Oct 30 01:36:23.287405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287190 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" event={"ID":"e467827d-8014-468c-a4e3-df6448b8ff15","Type":"ContainerDied","Data":"8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3"} Oct 30 01:36:23.287405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287230 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-tmp-dir\") pod \"e467827d-8014-468c-a4e3-df6448b8ff15\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " Oct 30 01:36:23.287405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287237 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" Oct 30 01:36:23.287405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287270 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-model-cache\") pod \"e467827d-8014-468c-a4e3-df6448b8ff15\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " Oct 30 01:36:23.287405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287302 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-home\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-home\") pod \"e467827d-8014-468c-a4e3-df6448b8ff15\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " Oct 30 01:36:23.287405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287161 4517 generic.go:356] "Generic (PLEG): container finished" podID="e467827d-8014-468c-a4e3-df6448b8ff15" containerID="8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3" exitCode=0 Oct 30 01:36:23.287405 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287349 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-dshm\") pod \"e467827d-8014-468c-a4e3-df6448b8ff15\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " Oct 30 01:36:23.287842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287431 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e467827d-8014-468c-a4e3-df6448b8ff15-kube-api-access-7ntxp\" (UniqueName: \"kubernetes.io/projected/e467827d-8014-468c-a4e3-df6448b8ff15-kube-api-access-7ntxp\") pod \"e467827d-8014-468c-a4e3-df6448b8ff15\" (UID: \"e467827d-8014-468c-a4e3-df6448b8ff15\") " Oct 30 01:36:23.287842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287472 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" event={"ID":"e467827d-8014-468c-a4e3-df6448b8ff15","Type":"ContainerDied","Data":"b775c868687fc95915e9da498560ea790b334ab11150947bc0b0e5c10936393c"} Oct 30 01:36:23.287842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287242 4517 scope.go:122] "RemoveContainer" containerID="8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3" Oct 30 01:36:23.287842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287558 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-model-cache" pod "e467827d-8014-468c-a4e3-df6448b8ff15" (UID: "e467827d-8014-468c-a4e3-df6448b8ff15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:23.287842 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.287837 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:23.288526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.288491 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-home" pod "e467827d-8014-468c-a4e3-df6448b8ff15" (UID: "e467827d-8014-468c-a4e3-df6448b8ff15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:23.288754 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.288695 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-tmp-dir" pod "e467827d-8014-468c-a4e3-df6448b8ff15" (UID: "e467827d-8014-468c-a4e3-df6448b8ff15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:23.290014 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.289988 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e467827d-8014-468c-a4e3-df6448b8ff15-tls-certs" pod "e467827d-8014-468c-a4e3-df6448b8ff15" (UID: "e467827d-8014-468c-a4e3-df6448b8ff15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:23.290596 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.290568 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-dshm" pod "e467827d-8014-468c-a4e3-df6448b8ff15" (UID: "e467827d-8014-468c-a4e3-df6448b8ff15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:23.291112 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.291085 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e467827d-8014-468c-a4e3-df6448b8ff15-kube-api-access-7ntxp" pod "e467827d-8014-468c-a4e3-df6448b8ff15" (UID: "e467827d-8014-468c-a4e3-df6448b8ff15"). InnerVolumeSpecName "kube-api-access-7ntxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:23.308943 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.308908 4517 scope.go:122] "RemoveContainer" containerID="8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3" Oct 30 01:36:23.309334 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:36:23.309313 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3\": container with ID starting with 8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3 not found: ID does not exist" containerID="8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3" Oct 30 01:36:23.309430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.309347 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3"} err="failed to get container status \"8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3\": rpc error: code = NotFound desc = could not find container \"8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3\": container with ID starting with 8bc989be2daf0f36c1ad77a526f297dc60ff6237af74077cfac6ae4dec5ee1a3 not found: ID does not exist" Oct 30 01:36:23.388969 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.388927 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:23.388969 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.388961 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:23.388969 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.388969 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e467827d-8014-468c-a4e3-df6448b8ff15-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:23.388969 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.388979 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ntxp\" (UniqueName: \"kubernetes.io/projected/e467827d-8014-468c-a4e3-df6448b8ff15-kube-api-access-7ntxp\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:23.389277 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.388990 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e467827d-8014-468c-a4e3-df6448b8ff15-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:23.630379 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.630337 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt"] Oct 30 01:36:23.640490 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:23.640449 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt"] Oct 30 01:36:24.093384 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:24.093332 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-7e040d/simulator-single-main-7e040d-kserve-cdb6f9d5d-8m2dt" podUID="e467827d-8014-468c-a4e3-df6448b8ff15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.241:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:36:24.239825 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:24.239784 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e467827d-8014-468c-a4e3-df6448b8ff15" path="/var/lib/kubelet/pods/e467827d-8014-468c-a4e3-df6448b8ff15/volumes" Oct 30 01:36:26.654115 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.654074 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj"] Oct 30 01:36:26.654525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.654422 4517 cpu_manager.go:414] "containerMap: removing container" podUID="d4b2442c-fd44-4e42-84f2-2e0f0172dd95" containerName="revoke-keys" Oct 30 01:36:26.654525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.654433 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b2442c-fd44-4e42-84f2-2e0f0172dd95" containerName="revoke-keys" Oct 30 01:36:26.654525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.654456 4517 cpu_manager.go:414] "containerMap: removing container" podUID="e467827d-8014-468c-a4e3-df6448b8ff15" containerName="main" Oct 30 01:36:26.654525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.654461 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e467827d-8014-468c-a4e3-df6448b8ff15" containerName="main" Oct 30 01:36:26.654525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.654512 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b2442c-fd44-4e42-84f2-2e0f0172dd95" containerName="revoke-keys" Oct 30 01:36:26.654525 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.654524 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="e467827d-8014-468c-a4e3-df6448b8ff15" containerName="main" Oct 30 01:36:26.657587 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.657564 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.672471 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.672439 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 01:36:26.682653 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.682620 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 01:36:26.684825 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.684792 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj"] Oct 30 01:36:26.721901 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.721865 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:26.722112 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.722050 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-28sms\"" Oct 30 01:36:26.722112 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.722088 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5feb53af-kserve-self-signed-certs\"" Oct 30 01:36:26.822047 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.822011 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph249\" (UniqueName: \"kubernetes.io/projected/6cdd34ab-26f7-499a-a3a3-572673ea25e9-kube-api-access-ph249\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.822271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.822065 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-model-cache\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.822271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.822156 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-home\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.822271 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.822199 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-dshm\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.822461 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.822347 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tls-certs\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.822461 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.822381 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tmp-dir\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.923426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.923321 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tls-certs\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.923426 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.923362 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tmp-dir\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.923689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.923491 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph249\" (UniqueName: \"kubernetes.io/projected/6cdd34ab-26f7-499a-a3a3-572673ea25e9-kube-api-access-ph249\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.923689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.923545 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-model-cache\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.923689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.923584 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-home\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.923689 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.923622 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-dshm\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.923928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.923789 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tmp-dir\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.923928 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.923911 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-model-cache\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.924043 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.923984 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-home\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.926074 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.926054 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-dshm\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.926178 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.926080 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tls-certs\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:26.982937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:26.982903 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph249\" (UniqueName: \"kubernetes.io/projected/6cdd34ab-26f7-499a-a3a3-572673ea25e9-kube-api-access-ph249\") pod \"e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:27.271539 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:27.271440 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:27.331389 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:27.331354 4517 scope.go:122] "RemoveContainer" containerID="e0b7f0f8f2b32efa1be3364e163eaf48cbd82855675fd5656d488fe2afb0e2b9" Oct 30 01:36:27.573981 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:27.573943 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj"] Oct 30 01:36:27.577383 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:36:27.577351 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cdd34ab_26f7_499a_a3a3_572673ea25e9.slice/crio-b104e89664b8711d7987415fe677d659dfbb46028452dcf5803c1320859221f5 WatchSource:0}: Error finding container b104e89664b8711d7987415fe677d659dfbb46028452dcf5803c1320859221f5: Status 404 returned error can't find the container with id b104e89664b8711d7987415fe677d659dfbb46028452dcf5803c1320859221f5 Oct 30 01:36:28.314218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:28.314181 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" event={"ID":"6cdd34ab-26f7-499a-a3a3-572673ea25e9","Type":"ContainerStarted","Data":"fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837"} Oct 30 01:36:28.314218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:28.314222 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" event={"ID":"6cdd34ab-26f7-499a-a3a3-572673ea25e9","Type":"ContainerStarted","Data":"b104e89664b8711d7987415fe677d659dfbb46028452dcf5803c1320859221f5"} Oct 30 01:36:28.353954 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:28.353890 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" podStartSLOduration=2.353876005 podStartE2EDuration="2.353876005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:28.352898296 +0000 UTC m=+3726.265025438" watchObservedRunningTime="2026-10-30 01:36:28.353876005 +0000 UTC m=+3726.266003194" Oct 30 01:36:29.271925 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:29.271886 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:29.992961 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:29.992918 4517 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" podUID="39bbc5f2-3224-4415-96d4-5f120ff7cfc3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:36:30.795733 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:36:30.795370 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:36:37.271906 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:37.271795 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:37.284765 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:37.284729 4517 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:37.371552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:37.371509 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:36:38.788584 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:38.788556 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:36:38.889071 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:38.888972 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-plugins-config-volume\") pod \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\" (UID: \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\") " Oct 30 01:36:38.889248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:38.889077 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-kube-api-access-66c2h\") pod \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\" (UID: \"39bbc5f2-3224-4415-96d4-5f120ff7cfc3\") " Oct 30 01:36:38.889435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:38.889415 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-plugins-config-volume" pod "39bbc5f2-3224-4415-96d4-5f120ff7cfc3" (UID: "39bbc5f2-3224-4415-96d4-5f120ff7cfc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:38.897182 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:38.892453 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-kube-api-access-66c2h" pod "39bbc5f2-3224-4415-96d4-5f120ff7cfc3" (UID: "39bbc5f2-3224-4415-96d4-5f120ff7cfc3"). InnerVolumeSpecName "kube-api-access-66c2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:38.990253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:38.990211 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-kube-api-access-66c2h\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:38.990253 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:38.990243 4517 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bbc5f2-3224-4415-96d4-5f120ff7cfc3-plugins-config-volume\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:36:39.371086 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:39.371046 4517 generic.go:356] "Generic (PLEG): container finished" podID="39bbc5f2-3224-4415-96d4-5f120ff7cfc3" containerID="94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad" exitCode=0 Oct 30 01:36:39.371326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:39.371118 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" Oct 30 01:36:39.371326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:39.371119 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" event={"ID":"39bbc5f2-3224-4415-96d4-5f120ff7cfc3","Type":"ContainerDied","Data":"94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad"} Oct 30 01:36:39.371326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:39.371191 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6" event={"ID":"39bbc5f2-3224-4415-96d4-5f120ff7cfc3","Type":"ContainerDied","Data":"5b028b583f392e4c3ea7c2579c330e471ad04afc82a1e435cb509eb7f48437bb"} Oct 30 01:36:39.371326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:39.371222 4517 scope.go:122] "RemoveContainer" containerID="94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad" Oct 30 01:36:39.380634 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:39.380612 4517 scope.go:122] "RemoveContainer" containerID="94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad" Oct 30 01:36:39.381031 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:36:39.381010 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad\": container with ID starting with 94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad not found: ID does not exist" containerID="94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad" Oct 30 01:36:39.381083 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:39.381043 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad"} err="failed to get container status \"94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad\": rpc error: code = NotFound desc = could not find container \"94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad\": container with ID starting with 94a39a3dd48fc23f607f565b439bab9febc0eb58287490f06514170fe5662aad not found: ID does not exist" Oct 30 01:36:39.418172 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:39.418131 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6"] Oct 30 01:36:39.421572 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:39.421543 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e040d-5cb4fzrxt6"] Oct 30 01:36:40.233165 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:40.233121 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39bbc5f2-3224-4415-96d4-5f120ff7cfc3" path="/var/lib/kubelet/pods/39bbc5f2-3224-4415-96d4-5f120ff7cfc3/volumes" Oct 30 01:36:43.164060 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:43.164022 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:44.833885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:44.833701 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:45.469114 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:45.469074 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:51.368648 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:36:51.368609 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:00.759439 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:37:00.759396 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:37:12.591819 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:12.591783 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj"] Oct 30 01:37:12.653488 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:12.653448 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:16.007326 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:16.007282 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:21.035407 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:21.035369 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:26.031269 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:26.031230 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck"] Oct 30 01:37:26.040919 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:26.040882 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30f9d1f-nddck"] Oct 30 01:37:26.233514 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:26.233469 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f45205-5760-4953-8b48-1cd311a96e1a" path="/var/lib/kubelet/pods/25f45205-5760-4953-8b48-1cd311a96e1a/volumes" Oct 30 01:37:27.613593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.613528 4517 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" podUID="6cdd34ab-26f7-499a-a3a3-572673ea25e9" containerName="main" containerID="cri-o://fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837" gracePeriod=45 Oct 30 01:37:27.863369 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.863342 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:37:27.871400 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.871374 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-dshm\") pod \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " Oct 30 01:37:27.871537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.871431 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-model-cache\") pod \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " Oct 30 01:37:27.871537 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.871488 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdd34ab-26f7-499a-a3a3-572673ea25e9-kube-api-access-ph249\" (UniqueName: \"kubernetes.io/projected/6cdd34ab-26f7-499a-a3a3-572673ea25e9-kube-api-access-ph249\") pod \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " Oct 30 01:37:27.871657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.871551 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-home\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-home\") pod \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " Oct 30 01:37:27.871657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.871598 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tls-certs\") pod \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " Oct 30 01:37:27.871801 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.871740 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-model-cache" pod "6cdd34ab-26f7-499a-a3a3-572673ea25e9" (UID: "6cdd34ab-26f7-499a-a3a3-572673ea25e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:27.871801 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.871750 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tmp-dir\") pod \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\" (UID: \"6cdd34ab-26f7-499a-a3a3-572673ea25e9\") " Oct 30 01:37:27.871922 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.871878 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-home" pod "6cdd34ab-26f7-499a-a3a3-572673ea25e9" (UID: "6cdd34ab-26f7-499a-a3a3-572673ea25e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:27.872036 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.872012 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tmp-dir" pod "6cdd34ab-26f7-499a-a3a3-572673ea25e9" (UID: "6cdd34ab-26f7-499a-a3a3-572673ea25e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:27.872160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.872113 4517 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-home\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:37:27.872160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.872131 4517 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tmp-dir\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:37:27.872160 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.872141 4517 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-model-cache\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:37:27.873962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.873940 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tls-certs" pod "6cdd34ab-26f7-499a-a3a3-572673ea25e9" (UID: "6cdd34ab-26f7-499a-a3a3-572673ea25e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:27.873962 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.873954 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-dshm" pod "6cdd34ab-26f7-499a-a3a3-572673ea25e9" (UID: "6cdd34ab-26f7-499a-a3a3-572673ea25e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:27.874147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.873972 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdd34ab-26f7-499a-a3a3-572673ea25e9-kube-api-access-ph249" pod "6cdd34ab-26f7-499a-a3a3-572673ea25e9" (UID: "6cdd34ab-26f7-499a-a3a3-572673ea25e9"). InnerVolumeSpecName "kube-api-access-ph249". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:27.972987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.972951 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph249\" (UniqueName: \"kubernetes.io/projected/6cdd34ab-26f7-499a-a3a3-572673ea25e9-kube-api-access-ph249\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:37:27.972987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.972984 4517 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdd34ab-26f7-499a-a3a3-572673ea25e9-tls-certs\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:37:27.972987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:27.972997 4517 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdd34ab-26f7-499a-a3a3-572673ea25e9-dshm\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:37:28.672854 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:28.672817 4517 generic.go:356] "Generic (PLEG): container finished" podID="6cdd34ab-26f7-499a-a3a3-572673ea25e9" containerID="fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837" exitCode=0 Oct 30 01:37:28.673324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:28.672876 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" Oct 30 01:37:28.673324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:28.672882 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" event={"ID":"6cdd34ab-26f7-499a-a3a3-572673ea25e9","Type":"ContainerDied","Data":"fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837"} Oct 30 01:37:28.673324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:28.672988 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj" event={"ID":"6cdd34ab-26f7-499a-a3a3-572673ea25e9","Type":"ContainerDied","Data":"b104e89664b8711d7987415fe677d659dfbb46028452dcf5803c1320859221f5"} Oct 30 01:37:28.673324 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:28.673008 4517 scope.go:122] "RemoveContainer" containerID="fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837" Oct 30 01:37:28.681933 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:28.681906 4517 scope.go:122] "RemoveContainer" containerID="fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837" Oct 30 01:37:28.682285 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:37:28.682262 4517 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837\": container with ID starting with fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837 not found: ID does not exist" containerID="fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837" Oct 30 01:37:28.682359 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:28.682298 4517 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837"} err="failed to get container status \"fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837\": rpc error: code = NotFound desc = could not find container \"fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837\": container with ID starting with fbc95c3c46d17ee32a63a5a26a1085e072e68a5bef6da389e9275eb5ff521837 not found: ID does not exist" Oct 30 01:37:28.718381 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:28.718348 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj"] Oct 30 01:37:28.728833 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:28.728759 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5feb53af-kserve-677cb5b997-plxzj"] Oct 30 01:37:30.233030 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:30.232999 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cdd34ab-26f7-499a-a3a3-572673ea25e9" path="/var/lib/kubelet/pods/6cdd34ab-26f7-499a-a3a3-572673ea25e9/volumes" Oct 30 01:37:30.761610 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:37:30.761572 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:37:31.354152 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:31.354112 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:33.573124 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:33.573086 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:40.845737 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:37:40.845668 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:00.758838 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:38:00.758789 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:38:15.082014 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.081976 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts"] Oct 30 01:38:15.082503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.082374 4517 cpu_manager.go:414] "containerMap: removing container" podUID="6cdd34ab-26f7-499a-a3a3-572673ea25e9" containerName="main" Oct 30 01:38:15.082503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.082389 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdd34ab-26f7-499a-a3a3-572673ea25e9" containerName="main" Oct 30 01:38:15.082503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.082399 4517 cpu_manager.go:414] "containerMap: removing container" podUID="39bbc5f2-3224-4415-96d4-5f120ff7cfc3" containerName="payload-pre-processing" Oct 30 01:38:15.082503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.082404 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bbc5f2-3224-4415-96d4-5f120ff7cfc3" containerName="payload-pre-processing" Oct 30 01:38:15.082503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.082468 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdd34ab-26f7-499a-a3a3-572673ea25e9" containerName="main" Oct 30 01:38:15.082503 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.082475 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="39bbc5f2-3224-4415-96d4-5f120ff7cfc3" containerName="payload-pre-processing" Oct 30 01:38:15.085586 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.085565 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.109022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.108986 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:38:15.109022 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.108989 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:38:15.109294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.109083 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:38:15.109294 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.109101 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:38:15.110605 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.110580 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts"] Oct 30 01:38:15.233769 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.233699 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z7zts\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.234000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.233797 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrgj\" (UniqueName: \"kubernetes.io/projected/39dea1e9-1464-4fc0-8308-c835aca18f23-kube-api-access-tkrgj\") pod \"maas-api-revoke-sub-f03629db-z7zts\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.234000 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.233932 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z7zts\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.335273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.335173 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z7zts\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.335273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.335224 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrgj\" (UniqueName: \"kubernetes.io/projected/39dea1e9-1464-4fc0-8308-c835aca18f23-kube-api-access-tkrgj\") pod \"maas-api-revoke-sub-f03629db-z7zts\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.335273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.335260 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z7zts\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.335898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.335877 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z7zts\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.337773 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.337757 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z7zts\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.357075 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.357034 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrgj\" (UniqueName: \"kubernetes.io/projected/39dea1e9-1464-4fc0-8308-c835aca18f23-kube-api-access-tkrgj\") pod \"maas-api-revoke-sub-f03629db-z7zts\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.398593 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.398553 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:15.694068 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.693963 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts"] Oct 30 01:38:15.698224 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:38:15.698195 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39dea1e9_1464_4fc0_8308_c835aca18f23.slice/crio-3c63cd7767001e67f0bbc13c9de5a4dc32a105fc622a440184e0db7d07059e72 WatchSource:0}: Error finding container 3c63cd7767001e67f0bbc13c9de5a4dc32a105fc622a440184e0db7d07059e72: Status 404 returned error can't find the container with id 3c63cd7767001e67f0bbc13c9de5a4dc32a105fc622a440184e0db7d07059e72 Oct 30 01:38:15.959945 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:15.959910 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" event={"ID":"39dea1e9-1464-4fc0-8308-c835aca18f23","Type":"ContainerStarted","Data":"3c63cd7767001e67f0bbc13c9de5a4dc32a105fc622a440184e0db7d07059e72"} Oct 30 01:38:16.966239 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:16.966203 4517 generic.go:356] "Generic (PLEG): container finished" podID="39dea1e9-1464-4fc0-8308-c835aca18f23" containerID="aff19cd0e4f7da6760cc258b808c8e69deb4826d23a86d80dc81cf3e2d7eea39" exitCode=0 Oct 30 01:38:16.966665 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:16.966297 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" event={"ID":"39dea1e9-1464-4fc0-8308-c835aca18f23","Type":"ContainerDied","Data":"aff19cd0e4f7da6760cc258b808c8e69deb4826d23a86d80dc81cf3e2d7eea39"} Oct 30 01:38:18.106894 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.106867 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:18.262218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.262130 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-xks-ca\") pod \"39dea1e9-1464-4fc0-8308-c835aca18f23\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " Oct 30 01:38:18.262218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.262185 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39dea1e9-1464-4fc0-8308-c835aca18f23-kube-api-access-tkrgj\" (UniqueName: \"kubernetes.io/projected/39dea1e9-1464-4fc0-8308-c835aca18f23-kube-api-access-tkrgj\") pod \"39dea1e9-1464-4fc0-8308-c835aca18f23\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " Oct 30 01:38:18.262218 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.262205 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-openshift-ca\") pod \"39dea1e9-1464-4fc0-8308-c835aca18f23\" (UID: \"39dea1e9-1464-4fc0-8308-c835aca18f23\") " Oct 30 01:38:18.262632 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.262610 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-openshift-ca" pod "39dea1e9-1464-4fc0-8308-c835aca18f23" (UID: "39dea1e9-1464-4fc0-8308-c835aca18f23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:18.264411 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.264387 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-xks-ca" pod "39dea1e9-1464-4fc0-8308-c835aca18f23" (UID: "39dea1e9-1464-4fc0-8308-c835aca18f23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:18.264495 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.264435 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dea1e9-1464-4fc0-8308-c835aca18f23-kube-api-access-tkrgj" pod "39dea1e9-1464-4fc0-8308-c835aca18f23" (UID: "39dea1e9-1464-4fc0-8308-c835aca18f23"). InnerVolumeSpecName "kube-api-access-tkrgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:18.363459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.363421 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.363459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.363453 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkrgj\" (UniqueName: \"kubernetes.io/projected/39dea1e9-1464-4fc0-8308-c835aca18f23-kube-api-access-tkrgj\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.363459 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.363468 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dea1e9-1464-4fc0-8308-c835aca18f23-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.976201 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.976164 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" event={"ID":"39dea1e9-1464-4fc0-8308-c835aca18f23","Type":"ContainerDied","Data":"3c63cd7767001e67f0bbc13c9de5a4dc32a105fc622a440184e0db7d07059e72"} Oct 30 01:38:18.976390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.976224 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c63cd7767001e67f0bbc13c9de5a4dc32a105fc622a440184e0db7d07059e72" Oct 30 01:38:18.976390 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:18.976176 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z7zts" Oct 30 01:38:21.158065 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:21.158021 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:27.380554 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:27.380509 4517 scope.go:122] "RemoveContainer" containerID="3d261800a9e479e20d11ec7f56e4effd4cef4557fd74bbc7470d8d0809401310" Oct 30 01:38:30.396980 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.396941 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt"] Oct 30 01:38:30.397531 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.397484 4517 cpu_manager.go:414] "containerMap: removing container" podUID="39dea1e9-1464-4fc0-8308-c835aca18f23" containerName="revoke-keys" Oct 30 01:38:30.397531 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.397504 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39dea1e9-1464-4fc0-8308-c835aca18f23" containerName="revoke-keys" Oct 30 01:38:30.397657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.397625 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="39dea1e9-1464-4fc0-8308-c835aca18f23" containerName="revoke-keys" Oct 30 01:38:30.402226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.402191 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.420921 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.420886 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt"] Oct 30 01:38:30.428494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.428458 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:38:30.428494 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.428494 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:38:30.444937 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.444902 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:38:30.445158 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.445006 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:38:30.463626 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.463574 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v7qnt\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.463847 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.463776 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhhn\" (UniqueName: \"kubernetes.io/projected/fe99b8a9-e37e-4ed9-8651-2d8677689e50-kube-api-access-wnhhn\") pod \"maas-api-revoke-sub-0165d7d2-v7qnt\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.463847 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.463832 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v7qnt\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.564486 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.564442 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v7qnt\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.564744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.564518 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v7qnt\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.564744 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.564585 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhhn\" (UniqueName: \"kubernetes.io/projected/fe99b8a9-e37e-4ed9-8651-2d8677689e50-kube-api-access-wnhhn\") pod \"maas-api-revoke-sub-0165d7d2-v7qnt\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.565175 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.565148 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v7qnt\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.567210 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.567183 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v7qnt\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.586410 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.586368 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhhn\" (UniqueName: \"kubernetes.io/projected/fe99b8a9-e37e-4ed9-8651-2d8677689e50-kube-api-access-wnhhn\") pod \"maas-api-revoke-sub-0165d7d2-v7qnt\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.717157 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:30.717063 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:30.762135 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:38:30.762039 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:38:31.024731 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:31.024616 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt"] Oct 30 01:38:31.029000 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:38:31.028967 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe99b8a9_e37e_4ed9_8651_2d8677689e50.slice/crio-460dbc598e914de9cf17a75ecf2b2f729f56847aca08cb1eac0615cfa3d21d8d WatchSource:0}: Error finding container 460dbc598e914de9cf17a75ecf2b2f729f56847aca08cb1eac0615cfa3d21d8d: Status 404 returned error can't find the container with id 460dbc598e914de9cf17a75ecf2b2f729f56847aca08cb1eac0615cfa3d21d8d Oct 30 01:38:31.040457 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:31.040416 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" event={"ID":"fe99b8a9-e37e-4ed9-8651-2d8677689e50","Type":"ContainerStarted","Data":"460dbc598e914de9cf17a75ecf2b2f729f56847aca08cb1eac0615cfa3d21d8d"} Oct 30 01:38:32.046437 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:32.046397 4517 generic.go:356] "Generic (PLEG): container finished" podID="fe99b8a9-e37e-4ed9-8651-2d8677689e50" containerID="91c9da3fec76a92c0f6b616da3dcbd50bb65e1aab8f3eded37e0752d53a35d8b" exitCode=0 Oct 30 01:38:32.046437 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:32.046438 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" event={"ID":"fe99b8a9-e37e-4ed9-8651-2d8677689e50","Type":"ContainerDied","Data":"91c9da3fec76a92c0f6b616da3dcbd50bb65e1aab8f3eded37e0752d53a35d8b"} Oct 30 01:38:33.186820 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.186787 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:33.281248 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.281215 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-openshift-ca\") pod \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " Oct 30 01:38:33.281439 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.281299 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-xks-ca\") pod \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " Oct 30 01:38:33.281439 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.281336 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe99b8a9-e37e-4ed9-8651-2d8677689e50-kube-api-access-wnhhn\" (UniqueName: \"kubernetes.io/projected/fe99b8a9-e37e-4ed9-8651-2d8677689e50-kube-api-access-wnhhn\") pod \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\" (UID: \"fe99b8a9-e37e-4ed9-8651-2d8677689e50\") " Oct 30 01:38:33.281557 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.281531 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-openshift-ca" pod "fe99b8a9-e37e-4ed9-8651-2d8677689e50" (UID: "fe99b8a9-e37e-4ed9-8651-2d8677689e50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:33.283584 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.283554 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe99b8a9-e37e-4ed9-8651-2d8677689e50-kube-api-access-wnhhn" pod "fe99b8a9-e37e-4ed9-8651-2d8677689e50" (UID: "fe99b8a9-e37e-4ed9-8651-2d8677689e50"). InnerVolumeSpecName "kube-api-access-wnhhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:33.283738 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.283613 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-xks-ca" pod "fe99b8a9-e37e-4ed9-8651-2d8677689e50" (UID: "fe99b8a9-e37e-4ed9-8651-2d8677689e50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:33.382652 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.382543 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:38:33.382652 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.382584 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe99b8a9-e37e-4ed9-8651-2d8677689e50-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:38:33.382652 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:33.382599 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhhn\" (UniqueName: \"kubernetes.io/projected/fe99b8a9-e37e-4ed9-8651-2d8677689e50-kube-api-access-wnhhn\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:38:34.058430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:34.058385 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" event={"ID":"fe99b8a9-e37e-4ed9-8651-2d8677689e50","Type":"ContainerDied","Data":"460dbc598e914de9cf17a75ecf2b2f729f56847aca08cb1eac0615cfa3d21d8d"} Oct 30 01:38:34.058430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:34.058413 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v7qnt" Oct 30 01:38:34.058430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:34.058439 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="460dbc598e914de9cf17a75ecf2b2f729f56847aca08cb1eac0615cfa3d21d8d" Oct 30 01:38:41.289889 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.289842 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc"] Oct 30 01:38:41.290435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.290299 4517 cpu_manager.go:414] "containerMap: removing container" podUID="fe99b8a9-e37e-4ed9-8651-2d8677689e50" containerName="revoke-keys" Oct 30 01:38:41.290435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.290315 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe99b8a9-e37e-4ed9-8651-2d8677689e50" containerName="revoke-keys" Oct 30 01:38:41.290435 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.290377 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe99b8a9-e37e-4ed9-8651-2d8677689e50" containerName="revoke-keys" Oct 30 01:38:41.293544 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.293492 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.317170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.317121 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:38:41.317416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.317128 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:38:41.317416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.317137 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:38:41.317416 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.317228 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:38:41.319026 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.318969 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc"] Oct 30 01:38:41.350632 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.350591 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-qjvbc\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.350853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.350640 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-qjvbc\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.350853 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.350804 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2x9p\" (UniqueName: \"kubernetes.io/projected/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-kube-api-access-f2x9p\") pod \"maas-api-revoke-sub-6b19338f-qjvbc\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.451578 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.451532 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-qjvbc\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.451812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.451604 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2x9p\" (UniqueName: \"kubernetes.io/projected/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-kube-api-access-f2x9p\") pod \"maas-api-revoke-sub-6b19338f-qjvbc\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.451812 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.451643 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-qjvbc\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.452240 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.452215 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-qjvbc\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.454323 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.454298 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-qjvbc\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.474229 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.474195 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2x9p\" (UniqueName: \"kubernetes.io/projected/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-kube-api-access-f2x9p\") pod \"maas-api-revoke-sub-6b19338f-qjvbc\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.607058 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.606955 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:41.910885 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:41.910799 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc"] Oct 30 01:38:41.914637 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:38:41.914604 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ff0391_58dc_4eb1_bfd8_fb8efeb9d81c.slice/crio-8165658234bbcdbaf43db3e62750a926aeaa901366d7921f949d3f73213bb5cb WatchSource:0}: Error finding container 8165658234bbcdbaf43db3e62750a926aeaa901366d7921f949d3f73213bb5cb: Status 404 returned error can't find the container with id 8165658234bbcdbaf43db3e62750a926aeaa901366d7921f949d3f73213bb5cb Oct 30 01:38:42.096597 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:42.096550 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" event={"ID":"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c","Type":"ContainerStarted","Data":"8165658234bbcdbaf43db3e62750a926aeaa901366d7921f949d3f73213bb5cb"} Oct 30 01:38:43.103432 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:43.103397 4517 generic.go:356] "Generic (PLEG): container finished" podID="95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c" containerID="5df5b7c5364d2534d3ad87a165af509f7b6aef161276031777cc933d94a649cd" exitCode=0 Oct 30 01:38:43.103953 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:43.103492 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" event={"ID":"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c","Type":"ContainerDied","Data":"5df5b7c5364d2534d3ad87a165af509f7b6aef161276031777cc933d94a649cd"} Oct 30 01:38:44.236309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.236280 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:44.277513 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.277473 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-xks-ca\") pod \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " Oct 30 01:38:44.277733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.277565 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-openshift-ca\") pod \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " Oct 30 01:38:44.277733 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.277601 4517 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-kube-api-access-f2x9p\" (UniqueName: \"kubernetes.io/projected/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-kube-api-access-f2x9p\") pod \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\" (UID: \"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c\") " Oct 30 01:38:44.277999 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.277968 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-openshift-ca" pod "95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c" (UID: "95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:44.280044 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.280018 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-xks-ca" pod "95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c" (UID: "95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:44.280234 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.280213 4517 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-kube-api-access-f2x9p" pod "95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c" (UID: "95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c"). InnerVolumeSpecName "kube-api-access-f2x9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:44.378667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.378574 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-xks-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:38:44.378667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.378611 4517 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-maas-api-openshift-ca\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:38:44.378667 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:44.378625 4517 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2x9p\" (UniqueName: \"kubernetes.io/projected/95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c-kube-api-access-f2x9p\") on node \"ip-10-0-132-84.ec2.internal\" DevicePath \"\"" Oct 30 01:38:45.120654 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:45.120615 4517 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" Oct 30 01:38:45.120879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:45.120617 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qjvbc" event={"ID":"95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c","Type":"ContainerDied","Data":"8165658234bbcdbaf43db3e62750a926aeaa901366d7921f949d3f73213bb5cb"} Oct 30 01:38:45.120879 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:38:45.120796 4517 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8165658234bbcdbaf43db3e62750a926aeaa901366d7921f949d3f73213bb5cb" Oct 30 01:39:00.759792 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:39:00.759747 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:39:03.139021 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:03.138992 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-5xnrn_4795d4ba-cbf6-4aa0-b315-f4e283de1229/manager/0.log" Oct 30 01:39:03.628414 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:03.628376 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2g96n_0a1c725a-d75d-40d7-9f9f-685551855bdc/manager/0.log" Oct 30 01:39:04.154174 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:04.154147 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4rx7v_67e01f0d-253b-4fba-9fea-fe443de7b5ae/server/0.log" Oct 30 01:39:04.582314 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:04.582276 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-t54b4_33c577c2-0113-4326-bcf9-bed2991d85f1/manager/0.log" Oct 30 01:39:04.824553 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:04.824516 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-cscsr_3476b7bb-1d20-42e9-bdc7-db7d9e919569/copy-manifests/0.log" Oct 30 01:39:04.854964 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:04.854889 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-cscsr_3476b7bb-1d20-42e9-bdc7-db7d9e919569/manager/0.log" Oct 30 01:39:06.665462 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:06.665428 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-v7qnt_fe99b8a9-e37e-4ed9-8651-2d8677689e50/revoke-keys/0.log" Oct 30 01:39:07.402123 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:07.402091 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-qjvbc_95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c/revoke-keys/0.log" Oct 30 01:39:07.573521 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:07.573484 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab6d1875-zrnl8_a38d1807-0afc-40ba-89c8-a560181367ce/revoke-keys/0.log" Oct 30 01:39:08.128059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:08.128030 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-z7zts_39dea1e9-1464-4fc0-8308-c835aca18f23/revoke-keys/0.log" Oct 30 01:39:10.019127 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:10.019092 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-wqsz4_6dc8574e-42a3-4ad5-855d-83ae930a9846/manager/0.log" Oct 30 01:39:13.246170 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:13.246141 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l_eee3d8a3-129b-4bf7-bb8e-106edb9b69d5/istio-proxy/0.log" Oct 30 01:39:13.444694 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:13.444663 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2_f8c57672-5995-4696-ba27-e9d1dd3e46a1/istio-proxy/0.log" Oct 30 01:39:13.636176 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:13.636134 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2_08f94bdb-35af-4ac9-b852-68a5cd24634f/istio-proxy/0.log" Oct 30 01:39:21.272592 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:21.272553 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:26.722031 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:26.722000 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-5xnrn_4795d4ba-cbf6-4aa0-b315-f4e283de1229/manager/0.log" Oct 30 01:39:27.144188 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:27.144150 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2g96n_0a1c725a-d75d-40d7-9f9f-685551855bdc/manager/0.log" Oct 30 01:39:27.409079 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:27.408998 4517 scope.go:122] "RemoveContainer" containerID="02c76117ad9781eb6d9c6ceb3492d747e28271c7d6bb455deaa3f95e8148ac3b" Oct 30 01:39:27.601748 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:27.601699 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4rx7v_67e01f0d-253b-4fba-9fea-fe443de7b5ae/server/0.log" Oct 30 01:39:27.997243 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:27.997210 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-t54b4_33c577c2-0113-4326-bcf9-bed2991d85f1/manager/0.log" Oct 30 01:39:28.230841 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:28.230807 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-cscsr_3476b7bb-1d20-42e9-bdc7-db7d9e919569/copy-manifests/0.log" Oct 30 01:39:28.260371 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:28.260289 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-cscsr_3476b7bb-1d20-42e9-bdc7-db7d9e919569/manager/0.log" Oct 30 01:39:29.808663 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:29.808637 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-v7qnt_fe99b8a9-e37e-4ed9-8651-2d8677689e50/revoke-keys/0.log" Oct 30 01:39:30.528580 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:30.528550 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-qjvbc_95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c/revoke-keys/0.log" Oct 30 01:39:30.707552 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:30.707511 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab6d1875-zrnl8_a38d1807-0afc-40ba-89c8-a560181367ce/revoke-keys/0.log" Oct 30 01:39:30.759758 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:39:30.759694 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:39:31.235818 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:31.235784 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-z7zts_39dea1e9-1464-4fc0-8308-c835aca18f23/revoke-keys/0.log" Oct 30 01:39:32.776520 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:32.776446 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-wqsz4_6dc8574e-42a3-4ad5-855d-83ae930a9846/manager/0.log" Oct 30 01:39:35.954681 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:35.954647 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l_eee3d8a3-129b-4bf7-bb8e-106edb9b69d5/istio-proxy/0.log" Oct 30 01:39:36.143724 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:36.143673 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2_f8c57672-5995-4696-ba27-e9d1dd3e46a1/istio-proxy/0.log" Oct 30 01:39:36.329199 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:39:36.329165 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2_08f94bdb-35af-4ac9-b852-68a5cd24634f/istio-proxy/0.log" Oct 30 01:40:00.756844 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:40:00.756802 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:40:21.407949 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:21.407910 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:30.760105 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:40:30.760060 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:40:51.795771 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.795685 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:51.810430 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.810381 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8hpkt/must-gather-d7mdn"] Oct 30 01:40:51.810795 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.810781 4517 cpu_manager.go:414] "containerMap: removing container" podUID="95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c" containerName="revoke-keys" Oct 30 01:40:51.810871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.810796 4517 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c" containerName="revoke-keys" Oct 30 01:40:51.810871 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.810870 4517 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ff0391-58dc-4eb1-bfd8-fb8efeb9d81c" containerName="revoke-keys" Oct 30 01:40:51.814037 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.814018 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hpkt/must-gather-d7mdn" Oct 30 01:40:51.827109 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.827078 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hpkt\"/\"kube-root-ca.crt\"" Oct 30 01:40:51.837526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.837497 4517 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hpkt\"/\"openshift-service-ca.crt\"" Oct 30 01:40:51.837526 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.837505 4517 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8hpkt\"/\"default-dockercfg-mlw7p\"" Oct 30 01:40:51.854171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.854069 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/863843a6-1a32-40ec-bbbf-c09d36aef04f-must-gather-output\") pod \"must-gather-d7mdn\" (UID: \"863843a6-1a32-40ec-bbbf-c09d36aef04f\") " pod="openshift-must-gather-8hpkt/must-gather-d7mdn" Oct 30 01:40:51.854171 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.854142 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfks7\" (UniqueName: \"kubernetes.io/projected/863843a6-1a32-40ec-bbbf-c09d36aef04f-kube-api-access-dfks7\") pod \"must-gather-d7mdn\" (UID: \"863843a6-1a32-40ec-bbbf-c09d36aef04f\") " pod="openshift-must-gather-8hpkt/must-gather-d7mdn" Oct 30 01:40:51.854768 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.854748 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hpkt/must-gather-d7mdn"] Oct 30 01:40:51.930107 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.930076 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:51.955351 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.955314 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfks7\" (UniqueName: \"kubernetes.io/projected/863843a6-1a32-40ec-bbbf-c09d36aef04f-kube-api-access-dfks7\") pod \"must-gather-d7mdn\" (UID: \"863843a6-1a32-40ec-bbbf-c09d36aef04f\") " pod="openshift-must-gather-8hpkt/must-gather-d7mdn" Oct 30 01:40:51.955481 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.955403 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/863843a6-1a32-40ec-bbbf-c09d36aef04f-must-gather-output\") pod \"must-gather-d7mdn\" (UID: \"863843a6-1a32-40ec-bbbf-c09d36aef04f\") " pod="openshift-must-gather-8hpkt/must-gather-d7mdn" Oct 30 01:40:51.955811 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.955793 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/863843a6-1a32-40ec-bbbf-c09d36aef04f-must-gather-output\") pod \"must-gather-d7mdn\" (UID: \"863843a6-1a32-40ec-bbbf-c09d36aef04f\") " pod="openshift-must-gather-8hpkt/must-gather-d7mdn" Oct 30 01:40:51.977136 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:51.977104 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfks7\" (UniqueName: \"kubernetes.io/projected/863843a6-1a32-40ec-bbbf-c09d36aef04f-kube-api-access-dfks7\") pod \"must-gather-d7mdn\" (UID: \"863843a6-1a32-40ec-bbbf-c09d36aef04f\") " pod="openshift-must-gather-8hpkt/must-gather-d7mdn" Oct 30 01:40:52.128276 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:52.128186 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hpkt/must-gather-d7mdn" Oct 30 01:40:52.431281 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:52.431195 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hpkt/must-gather-d7mdn"] Oct 30 01:40:52.435417 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:40:52.435390 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod863843a6_1a32_40ec_bbbf_c09d36aef04f.slice/crio-d47b07260c0491f0607094e43114e968689da765a149d70acccc0b9bd79abee3 WatchSource:0}: Error finding container d47b07260c0491f0607094e43114e968689da765a149d70acccc0b9bd79abee3: Status 404 returned error can't find the container with id d47b07260c0491f0607094e43114e968689da765a149d70acccc0b9bd79abee3 Oct 30 01:40:52.437919 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:52.437893 4517 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:40:52.687821 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:52.687737 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hpkt/must-gather-d7mdn" event={"ID":"863843a6-1a32-40ec-bbbf-c09d36aef04f","Type":"ContainerStarted","Data":"d47b07260c0491f0607094e43114e968689da765a149d70acccc0b9bd79abee3"} Oct 30 01:40:53.698299 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:53.698256 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hpkt/must-gather-d7mdn" event={"ID":"863843a6-1a32-40ec-bbbf-c09d36aef04f","Type":"ContainerStarted","Data":"4d70012c3f5bfdc3b5d057bc6c748946a4d3588e19d783550089d8beae6a906d"} Oct 30 01:40:53.698795 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:53.698310 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hpkt/must-gather-d7mdn" event={"ID":"863843a6-1a32-40ec-bbbf-c09d36aef04f","Type":"ContainerStarted","Data":"502c9ef634fb48fb6ec6ba0fee7671d1633ed2b0bd1f23f38bd235f697aeafe4"} Oct 30 01:40:53.737944 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:53.737877 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8hpkt/must-gather-d7mdn" podStartSLOduration=2.01263218 podStartE2EDuration="2.737862539s" totalImagesPullingTime="725.230359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:53.737081447 +0000 UTC m=+3991.649208588" watchObservedRunningTime="2026-10-30 01:40:53.737862539 +0000 UTC m=+3991.649989679" Oct 30 01:40:56.040796 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:56.040766 4517 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8"] Oct 30 01:40:56.053531 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:56.053497 4517 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6d1875-zrnl8"] Oct 30 01:40:56.234315 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:56.234274 4517 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38d1807-0afc-40ba-89c8-a560181367ce" path="/var/lib/kubelet/pods/a38d1807-0afc-40ba-89c8-a560181367ce/volumes" Oct 30 01:40:56.623787 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:56.623751 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kprr7_4d0d2d6e-c723-4393-92e5-65c3e8b4f713/global-pull-secret-syncer/0.log" Oct 30 01:40:57.238678 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:57.238640 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" Oct 30 01:40:57.803052 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:40:57.802845 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-84.ec2.internal_e7678942b8a9336cefbe126b25b5ab55/haproxy/0.log" Oct 30 01:41:00.822891 ip-10-0-132-84 kubenswrapper[4517]: E1030 01:41:00.822848 4517 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lj5vr" containerName="extractor" Oct 30 01:41:03.935382 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:03.935352 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-wqsz4_6dc8574e-42a3-4ad5-855d-83ae930a9846/manager/0.log" Oct 30 01:41:06.637331 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.637283 4517 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n"] Oct 30 01:41:06.642548 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.642521 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.673130 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.673088 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n"] Oct 30 01:41:06.700657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.700555 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-podres\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.700657 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.700624 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-sys\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.700973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.700675 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-proc\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.700973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.700855 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-lib-modules\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.700973 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.700909 4517 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxlsl\" (UniqueName: \"kubernetes.io/projected/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-kube-api-access-gxlsl\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.803260 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.803215 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-proc\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.803598 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.803572 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-lib-modules\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.803919 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.803785 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxlsl\" (UniqueName: \"kubernetes.io/projected/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-kube-api-access-gxlsl\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.804139 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.804123 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-podres\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.804273 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.804243 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-podres\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.804470 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.803832 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-proc\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.804577 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.804257 4517 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-sys\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.804785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.804310 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-sys\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.804785 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.803895 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-lib-modules\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.827361 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.827320 4517 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxlsl\" (UniqueName: \"kubernetes.io/projected/b01fcf5b-97d4-4609-b0f5-e6c647eaac5a-kube-api-access-gxlsl\") pod \"perf-node-gather-daemonset-srw5n\" (UID: \"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:06.959240 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:06.959140 4517 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:07.285094 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:07.285056 4517 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n"] Oct 30 01:41:07.289099 ip-10-0-132-84 kubenswrapper[4517]: W1030 01:41:07.289063 4517 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb01fcf5b_97d4_4609_b0f5_e6c647eaac5a.slice/crio-97b4f48d98d79422727aae589046f06c85dc7a0733b83db1d0587cd7f2f2c945 WatchSource:0}: Error finding container 97b4f48d98d79422727aae589046f06c85dc7a0733b83db1d0587cd7f2f2c945: Status 404 returned error can't find the container with id 97b4f48d98d79422727aae589046f06c85dc7a0733b83db1d0587cd7f2f2c945 Oct 30 01:41:07.849575 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:07.849527 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" event={"ID":"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a","Type":"ContainerStarted","Data":"7d531d88ff2043e760555f301c00638783b199ce522362883bf8f6c52aaa6a99"} Oct 30 01:41:07.850147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:07.849596 4517 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" event={"ID":"b01fcf5b-97d4-4609-b0f5-e6c647eaac5a","Type":"ContainerStarted","Data":"97b4f48d98d79422727aae589046f06c85dc7a0733b83db1d0587cd7f2f2c945"} Oct 30 01:41:07.850147 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:07.849694 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:07.887586 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:07.887535 4517 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" podStartSLOduration=1.8875192379999999 podStartE2EDuration="1.887519238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:07.886413096 +0000 UTC m=+4005.798540237" watchObservedRunningTime="2026-10-30 01:41:07.887519238 +0000 UTC m=+4005.799646378" Oct 30 01:41:07.970013 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:07.969947 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-khckl_b18504a9-4418-4d89-a4f8-70e909d507f5/node-exporter/0.log" Oct 30 01:41:08.087059 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:08.087027 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-khckl_b18504a9-4418-4d89-a4f8-70e909d507f5/kube-rbac-proxy/0.log" Oct 30 01:41:08.210967 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:08.210893 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-khckl_b18504a9-4418-4d89-a4f8-70e909d507f5/init-textfile/0.log" Oct 30 01:41:08.377449 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:08.377413 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:08.493250 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:08.493130 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:10.167226 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:10.167161 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-b2fbs_50c14243-71e3-4c07-986a-6a25bbffa270/prometheus-operator-admission-webhook/0.log" Oct 30 01:41:13.869867 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:13.869834 4517 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-srw5n" Oct 30 01:41:14.622961 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:14.622926 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65c48c844f-47cwx_256e8329-af55-4138-8f59-f3caf10b881b/console/0.log" Oct 30 01:41:14.755611 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:14.755574 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-xmbzq_9c39f3ef-0c7f-49d6-a59f-639ac15dcb00/download-server/0.log" Oct 30 01:41:18.877898 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:18.877860 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tg442_b43fb340-5191-4f45-b6d9-9916afc9a2fd/dns/0.log" Oct 30 01:41:19.033987 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:19.033946 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tg442_b43fb340-5191-4f45-b6d9-9916afc9a2fd/kube-rbac-proxy/0.log" Oct 30 01:41:19.163193 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:19.163108 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-62mqm_16d03985-22a9-42a8-a195-638cb40e9e2c/dns-node-resolver/0.log" Oct 30 01:41:20.623878 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:20.623832 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-76467d86df-tb287_e9175d6a-2922-45c2-9b76-d49d7a775e3d/registry/0.log" Oct 30 01:41:20.807516 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:20.807483 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l_eee3d8a3-129b-4bf7-bb8e-106edb9b69d5/istio-proxy/0.log" Oct 30 01:41:20.872630 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:20.872589 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2_f8c57672-5995-4696-ba27-e9d1dd3e46a1/istio-proxy/0.log" Oct 30 01:41:20.993998 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:20.993889 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2_08f94bdb-35af-4ac9-b852-68a5cd24634f/istio-proxy/0.log" Oct 30 01:41:21.061869 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:21.061831 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zbzqk_1fa5c5ce-6d41-4294-ae89-c5f54b35dfe9/node-ca/0.log" Oct 30 01:41:21.565463 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:21.565427 4517 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jz5dq_3e7f74c1-965a-4ccc-a8f0-2dbd1770908e/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:24.894933 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:24.894900 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-hjz7l_eee3d8a3-129b-4bf7-bb8e-106edb9b69d5/istio-proxy/0.log" Oct 30 01:41:25.021309 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:25.021276 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nv6m2_f8c57672-5995-4696-ba27-e9d1dd3e46a1/istio-proxy/0.log" Oct 30 01:41:25.168873 ip-10-0-132-84 kubenswrapper[4517]: I1030 01:41:25.168781 4517 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nzbt2_08f94bdb-35af-4ac9-b852-68a5cd24634f/istio-proxy/0.log"